Author: Pan Shengwei (Ten Sleeps)

The profile

Full-link gray scale is one of the most core functions of microservices, and it has always been a necessary function for customers on the cloud in the in-depth process of microservices. Full-link gray scale involves many technologies and scenarios. If enterprises on the cloud implement it by themselves, they need to spend a lot of labor cost to expand and maintain it.

Technical areas involved

  • RPC: Indicates the route between micro services

    • Java system involves Spring Cloud, Apache Dubbo mainstream micro-service framework, and multi-language Service Mesh
    • DEBUG online traffic in the scenario of end-to-end cloud interconnection. Local The local service is registered with the registry through the jumper. After the online traffic meets the routing rules, the debugging is performed on the instance of the local service. The traffic that does not meet the routing rules is routed to the online instance

  • ​MQ​

    • In the full-link manslaughter scenario, the manslaughter traffic sends messages to the shadow Topic, and the manslaughter traffic only subscribles to the shadow Topic
    • In traffic isolation/full-link grayscale scenario, online traffic subscribes to online messages using the same Topic, and isolated traffic subscribes to grayscale messages only

  • ​Database​

    • In a full-link pressure test scenario, the pressure test traffic data is stored in the database to the shadow table
    • Prohibit database operations in high availability cut flow scenarios. In unitary mode, traffic does not have unit labels and database operations are prohibited

  • ​Redis​

    • In the full-link pressure test scenario, the pressure test traffic cache is transferred to the shadow KEY
    • Cache operations are prohibited in high availability cut flow scenarios. In unitary mode, traffic does not have a unit label, and caching is prohibited

  • Distributed task scheduling

    • For task scheduling, tasks submitted by grayscale environment are scheduled to be executed on machines in grayscale environment

  • The front end

    • The page information displayed by different customers is inconsistent

  • observability

    • Monitor traffic direction through observability to check traffic escape

MSE full link gray scale solution

At present, MSE Service Governance Professional edition provides a complete productization of the full link gray scale solution, covering RPC, MQ, observability and most of the scenarios. As long as your architecture is based on Spring Cloud or Dubbo framework, your application can play with enterprise-class full-link grayscale functionality without upgrading or changing a line of code.

  • Full link isolated traffic swimlane

1) ‘dye’ the required flow by setting the flow rule, and the ‘dye’ flow will be routed to the grayscale machine.

2) Grayscale flow carries grayscale scale to the downstream to form a grayscale exclusive environmental flow swimlane. The application without grayscale environment will choose the unmarked baseline environment by default.

  • An end-to-end stable baseline environment

The unmarked application belongs to the stable version of the baseline application, that is, the stable online environment. When we will publish the corresponding grayscale version of the code, we can then configure rules to direct the introduction of specific online traffic and control the risk of grayscale code.

  • Flow one-key dynamic flow cutting

After traffic rules are customized, you can stop, add, delete, modify, and check traffic rules with one click as required. The rules take effect in real time. Gray drainage is more convenient.

  • Observable capacity

Single application observability at lane level

At the same time, it has the observation capability of the whole link application, and can observe whether the traffic escapes from the global perspective. It’s easy to see if there’s any ash.

  • Low-cost access, implemented based on Java Agent technology without modifying a line of service code

MSE microservice governance capability is implemented based on Java Agent bytecode enhancement technology, which seamlessly supports all Spring Cloud and Dubbo versions available in the market for the past five years. Users can use it without changing a line of code, without changing the existing architecture of the business, and can go up and down at any time without binding. You only need to enable MSE Microservice Governance Professional edition to configure online and take effect in real time.

  • With lossless up and down the ability to make the release more silky

After MSE microservice management is enabled, applications can be connected online or offline without damage. In scenarios such as publishing, rolling back, capacity expansion, and capacity reduction under heavy traffic, traffic is not damaged.

How to use MSE full link Gray scale

Next, we will demonstrate the capability of full link gray scale. The application architecture we use consists of Ingress-Nginx and the back-end microservice architecture (Spring Cloud). The back-end call link has 3 hops, shopping cart (A), transaction center (B), inventory center (C), Clients access back-end services through clients or H5 pages, and they do service discovery through the Nacos registry.

The preparatory work

Open MSE microservice Governance professional edition

Log in to the MSE Management center console. If you have not enabled the MSE microservice management, open the Professional version as prompted. If you have already started the MSE Microservice Governance Foundation edition, follow the instructions on the right of the Overview page to upgrade to the Professional edition.

Install the Ingress-nginx component

  1. Log in to the Container Services console [1].
  2. In the navigation tree on the left, choose Market > Application Catalog.
  3. inThe application directoryEnter ack-ingress-nginx in the search box and clickIcon, and then click components.
  4. On the details page, select the component’s namespace as Kube-System, and click Create. After the installation is complete, if the Ack-ingress-nginx-default-controller application is displayed in namespace kube-system, the installation is successful.

Deploy the Demo application

Save the following files to ingress-gray-demo-deployment-set.yaml and execute kubectl apply -f ingress-gray-demo-deployment-set.yaml to deploy the application, Here we will deploy three applications A, B, and C, each with A baseline version and A grayscale version.

ApiVersion: apps/v1 kind: Deployment metadata: name: spring-cloud-a spec: replicas: 2 selector: matchLabels: app: spring-cloud-a template: metadata: annotations: msePilotCreateAppName: spring-cloud-a labels: app: Spring-cloud -a spec: containers: - env: -name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openJDK /jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT imagePullPolicy: Always name: spring-cloud-a ports: - containerPort: 20001 livenessProbe: tcpSocket: port: 20001 initialDelaySeconds: 10 periodSeconds: 30 # A apiVersion: apps/ V1 kind: Deployment metadata: name: Spring-cloud-a-new spec: replicas: 2 selector: matchLabels: app: spring-cloud-a-new strategy: template: metadata: annotations: alicloud.service.tag: gray msePilotCreateAppName: spring-cloud-a labels: app: spring-cloud-a-new spec: containers: -env: -name: JAVA_HOME value: /usr/lib/jvm/java-1.8 -openJDK /jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT imagePullPolicy: Always name: spring-cloud-a-new ports: - containerPort: 20001 livenessProbe: tcpSocket: port: 20001 initialDelaySeconds: 10 periodSeconds: 30 # B Base version -- apiVersion: apps/v1 kind: Deployment metadata: name: Spring-cloud-b spec: replicas: 2 selector: matchLabels: app: spring-cloud-b strategy: template: metadata: annotations: msePilotCreateAppName: spring-cloud-b labels: app: spring-cloud-b spec: containers: - env: - name: JAVA_HOME value: The/usr/lib/JVM/Java - 1.8 - its/jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT imagePullPolicy: Always name: spring-cloud-b ports: - containerPort: 8080 livenessProbe: tcpSocket: port: 20002 initialDelaySeconds: 10 periodSeconds: 30 # B GrayVersion -- apiVersion: apps/v1 kind: Deployment metadata: name: Spring-cloud-b-new spec: replicas: 2 selector: matchLabels: app: spring-cloud-b-new template: metadata: annotations: alicloud.service.tag: gray msePilotCreateAppName: spring-cloud-b labels: app: spring-cloud-b-new spec: containers: - env: - name: JAVA_HOME value: / usr/lib/JVM/Java - 1.8 - its/jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT imagePullPolicy: Always name: spring-cloud-b-new ports: - containerPort: 8080 livenessProbe: tcpSocket: port: 20002 initialDelaySeconds: 10 periodSeconds: 30 # C Base version -- apiVersion: apps/v1 kind: Deployment metadata: name: Spring-cloud-c spec: replicas: 2 selector: matchLabels: app: spring-cloud-c template: metadata: annotations: msePilotCreateAppName: spring-cloud-c labels: app: spring-cloud-c spec: containers: - env: - name: JAVA_HOME value: The/usr/lib/JVM/Java - 1.8 - its/jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT imagePullPolicy: Always name: spring-cloud-c ports: - containerPort: 8080 livenessProbe: tcpSocket: port: 20003 initialDelaySeconds: 10 periodSeconds: 30 # C GrayVersion -- apiVersion: apps/v1 kind: Deployment metadata: name: Spring-cloud-c-new spec: replicas: 2 selector: matchLabels: app: spring-cloud-c-new template: metadata: annotations: alicloud.service.tag: gray msePilotCreateAppName: spring-cloud-c labels: app: spring-cloud-c-new spec: containers: - env: - name: JAVA_HOME value: / usr/lib/JVM/Java - 1.8 - its/jre image: Registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT imagePullPolicy: IfNotPresent name: spring-cloud-c-new ports: - containerPort: 8080 livenessProbe: tcpSocket: port: 20003 initialDelaySeconds: 10 periodSeconds: 30 # Nacos Server --- apiVersion: apps/v1 kind: Deployment metadata: name: nacos-server spec: replicas: 1 selector: matchLabels: app: nacos-server template: metadata: labels: app: nacos-server spec: containers: - env: - name: MODE value: standalone image: nacos/nacos-server:latest imagePullPolicy: Always name: Nacos-server dnsPolicy: ClusterFirst restartPolicy: Always # nacos Server Service configuration -- apiVersion: v1 kind: Service metadata: name: nacos-server spec: ports: - port: 8848 protocol: TCP targetPort: 8848 selector: app: nacos-server type: ClusterIPCopy the code

For entry application A, two K8S services are configured. Spring-cloud-a-base corresponds to the Base version of A, and spring-cloud-a-Gray corresponds to the Gray version of A.

apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-base
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a

---
apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-gray
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a-new
Copy the code

Quickly build full link gray scale capability

  • Swimlanes are defined as a set of isolation environments for the same version application. Only the request traffic that meets the flow control routing rules is routed to the marking application in the corresponding swimlane. An application can belong to multiple lanes, and a lane can contain multiple applications. The relationship between applications and lanes is many-to-many.

  • Lane group: a collection of swimming lanes. Lane groups are mainly used to distinguish between teams or scenarios.

Log in to the MSE Governance Center console [2] and go to Microservice Governance Center > Full Link Gray scale

As you can see we need to implement the capabilities described above, we only need to do two steps, create a swimlane group and create a swimlane

Create a swimlane group

Click the button to create A swimlane group and select the back-end microservice application involved in our swimlane group, which is A, B and C according to the above demo

​​

Create a lane

At the top of the full-link gray scale page, select the same micro-service space that was used to create the swimlane group, and click at the bottom to create the first shunt swimlane. Note that applications that add full-link traffic control no longer support functions such as canary advertisement and label routing.

​​

According to product Step, we need to create a swimlane name, configure the application label, select the associated label of the swimlane, and configure routing rules for Ingress on the ACK console.

Create a completed swimlane

Look at the swimlanes. There are two modes

  • The view mode

  • Editable mode

Ingress rules for entry

Configure the Ingress rule for an entry. Access www.base.com* to route to the Base version of application A, and access www.gray.com* to route to the Gray version of application A.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-base
spec:
  rules:
  - host: www.base.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-base
          servicePort: 20001
        path: /

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-gray
spec:
  rules:
  - host: www.gray.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-gray
          servicePort: 20001
        path: /
Copy the code

Verify that characteristic traffic is routed to the target application

  1. results

Access *www.base.com* route to baseline environment

Curl -h "Host:www.base.com" http://} {ingress - IP/a [172.18.144.155] - > B [172.18.144.120] - > [172.18.144.79] % CCopy the code

At this point, access to *www.gray.com* is routed to the grayscale environment

Curl -h "Host:www.gray.com" http://{ingress-ip}/a Agray[172.18.144.160] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%Copy the code
  1. View the traffic monitoring diagram of the marking application.

Select the target swimlane group on the full-link gray page. You can select an application to display the traffic view

  1. View all application monitoring views.

In addition to viewing the monitoring view of a single application, you can also view the monitoring view of all applications in a swimlane group. You can obtain more useful information by comparing and analyzing the monitoring charts of all applications.

  • You can see which applications are being called at the same time.
  • Analyze the flow escape problem and judge the escape object.

conclusion

The full-link grayscale productibility capability of MSE service governance is still evolving. Currently, we support MQ, RPC, observables, etc., and more scenarios such as XXL-Job will be supported in the future. At present, we have realized multi-language microservice governance on MSE [3], end-cloud interconnection of microservices using Cloud Toolkit [4], full-link gray scale based on Ingress-Nginx gateway [5], full-link gray scale based on MSE Cloud native gateway [6] , realized full-link gray scale based on self-built Spring Cloud Gateway or Zuul Gateway [7], realized full-link gray scale based on RocketMQ version of message queue [8], and realized Canary publishing by Constructing CI/CD by Jenkins [9] , micro-service agile development best practices [10] and other complete solutions related to full-link gray scale. With the increase of user scenarios and practices, our solutions will continue to iterate and enrich.

Typical cases

The electricity technology

MSE service governance helps our system quickly realize the full-link grayscale capability at a very low cost and in a non-invasive manner, which further improves the stability of our system and makes us feel more secure in the iteration of our new requirements.

— Tang Changzheng, technology architect of Telex

Since 2014, The company has entered the field of shared charging, defined and created the industry, belonging to the earliest shared charging enterprise in the industry. The main business covers charging bank self-service rental, customized shopping mall navigation machine development, advertising display equipment and advertising communication services. Call technology has the industry three-dimensional product line, small and medium-sized cabinets and desktop type, more than 90% of the country’s cities to achieve business services landing, registered users more than 200 million people, to achieve the full scene of user needs.

The whole link grayscale falls to the ground

The business architecture of the call is as follows, the top layer is the user interface such as mobile terminal, the self-built Nginx gateway as the access layer, the service layer is a variety of services, using Spring Cloud and Dubbo as the service framework.

​​

The architecture of the whole link gray landing is as follows:

Configure the traffic shunt configuration in the Nginx layer, 10% of the traffic enters the grayscale environment, and 90% of the traffic enters the formal online environment without marking. Then the traffic passing through the grayscale environment will be automatically colored by MSE with the color of the corresponding environment, so as to carry out the full-link grayscale routing and ensure that the traffic is closed loop in the grayscale environment. If there are no machines with gray scale environment, such as online machines in the payment center, then the traffic will go through the online environment. When there are machines with gray scale environment in our data center, the gray scale traffic will return to the gray scale environment in the data center.

The tail

MSE’s full-link grayscale capability continues to expand and iterate with the deepening of customer scenarios. Only products polished by customers will be more and more enduring and fresh. Welcome to try fresh experience.

A link to the

[1] Container Services console

​​https://cs.console.aliyun.com/#/authorize​​

[2] MSE Governance Center console

​​https://mse.console.aliyun.com/?spm=a2c4g.11186623.2.13.f90a6a60WiEx0N#/auth​​

[3] How to implement multilingual microservice governance on MSE

​​https://help.aliyun.com/document_detail/184289.html​​

[4] Cloud Toolkit is used to realize end-cloud interconnection of microservices

​​https://help.aliyun.com/document_detail/196920.html​​

[5] Ingress-Nginx gateway to achieve full link gray scale

​​https://help.aliyun.com/document_detail/347790.html​​

[6] Full link gray scale based on MSE cloud native gateway

​​https://help.aliyun.com/document_detail/359851.html​​

[7] Achieve full-link gray scale based on self-built Spring Cloud Gateway or Zuul Gateway

​​https://help.aliyun.com/document_detail/359858.html​​

[8] Full link gray scale implementation based on RocketMQ version of message queue

​​https://help.aliyun.com/document_detail/397318.html​​

[9] Jenkins builds CI/CD for Canary release

​​https://help.aliyun.com/document_detail/384436.html​​

[10] Best practices for microservices Agile development

​​https://help.aliyun.com/document_detail/397319.html​​

Click here for more details