background

In some cases, we are using Kubernetes as a cloud platform for business applications. We want to implement blue-green deployment of applications for iterative application versions. Lstio is too heavy and complex, and it is positioned for flow control and grid governance. Ingress-nginx introduced the Canary feature in version 0.21, which allows you to configure multiple versions of the application for gateway entry, using annotations to control traffic allocation for multiple back-end services

Ingress-nginx -Annotation Canary

If you want to enable the Canary function, the first to set up nginx. Ingress. Kubernetes. IO/Canary: “true”, then can enable the following comments to configure Canary

  • nginx.ingress.kubernetes.io/canary-weightThe percentage of requests to the service specified in Canary ingress, an integer ranging from 0 to 100, which determines the approximate percentage of traffic allocated to the back-end S service specified in Canary ingress
  • nginx.ingress.kubernetes.io/canary-by-headerThe request Header-based traffic splitting is suitable for gray scale distribution or A/B testing. When the hearder value is set to always, the request traffic is always allocated to the Canary port. When the hearder value is set to never, the request traffic is not allocated to the Canary port. Other hearder values are ignored and request traffic is allocated to other rules by priority
  • nginx.ingress.kubernetes.io/canary-by-header-valueThis configuration needs to be matchednginx.ingress.kubernetes.io/canary-by-headerUsed together when the hearder key and value in the request andnginx.ingress.kubernetes.io/canary-by-header nginx.ingress.kubernetes.io/canary-by-header-valueWhen matched, request traffic is allocated to the Canary Ingress entry, and any other hearder values are ignored and assigned to other rules by priority
  • nginx.ingress.kubernetes.io/canary-by-cookieThis configuration is based on cookie traffic segmentation, which is also suitable for grayscale distribution or A/B testing. When cookie is set to always, the requested traffic is routed to Canary Ingress, but when cookie is set to never, the requested traffic is not routed to Canary. Other values are ignored and request traffic is allocated to other rules by priority

Canary rules are sorted in priority as follows: canary-by-header – > canary-by-cookie – > canary-weight

1. Small-scale version testing based on weights

  • Format files for v1 version
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: echoserverv1
  name: echoserverv1
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv1
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name:  echoserverv1
  namespace: echoserver
spec:
  selector:
    name:  echoserverv1
  type:  ClusterIP
  ports:
  - name:  echoserverv1
    port:  8080
    targetPort:  8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  echoserverv1
  namespace: echoserver
  labels:
    name:  echoserverv1
spec:
  template:
    metadata:
      labels:
        name:  echoserverv1
    spec:
      containers:
      - image:  Mirrorgooglecontainers/echoserver: 1.10
        name:  echoserverv1 
        ports:
        - containerPort:  8080
          name:  echoserverv1
Copy the code
  • View resources created in v1
$ [K8sSj] kubectl get pod,service,ingress -n echoserver NAME READY STATUS RESTARTS AGE pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h NAME TYPE cluster-ip external-ip PORT(S) AGE service/ EchoServerv1 ClusterIP 10.99.68.72 < None > 8080/TCP 24h NAME HOSTS ADDRESS PORTS AGE ingress.extensions/echoserverv1 echo.chulinx.com 80 24hCopy the code
  • Accessing the v1 service, you can see that all 10 requests are going to a POD, which is the v1 version of the service
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Copy the code
  • Create a v2 service

We turned on the Canary function and set the weight of v2 version to 50%. This percentage does not accurately divide requests evenly between the two versions of services, but floats around 50%

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
  labels:
    app: echoserverv2
  name: echoserverv2
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv2
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name:  echoserverv2
  namespace: echoserver
spec:
  selector:
    name:  echoserverv2
  type:  ClusterIP
  ports:
  - name:  echoserverv2
    port:  8080
    targetPort:  8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  echoserverv2
  namespace: echoserver
  labels:
    name:  echoserverv2
spec:
  template:
    metadata:
      labels:
        name:  echoserverv2
    spec:
      containers:
      - image:  Mirrorgooglecontainers/echoserver: 1.10
        name:  echoserverv2 
        ports:
        - containerPort:  8080
          name:  echoserverv2
Copy the code
  • Look again at the created resource
$ [K8sSj] kubectl get pod,service,ingress -n echoserver NAME READY STATUS RESTARTS AGE pod/echoserverv1-657b966cb5-7grqs  1/1 Running 0 24h pod/echoserverv2-856bb5758-f9tqN 1/1 Running 0 4s NAME TYPE cluster-ip external-ip PORT(S) AGE service/ EchoServerv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h service/ ECHOServerv2 ClusterIP 10.111.103.170 < None > 8080/TCP 4S NAME HOSTS ADDRESS PORTS AGE ingress.extensions/echoserverv1 echo.chulinx.com 80 24h ingress.extensions/echoserverv2 echo.chulinx.com 80 4sCopy the code
  • Access to the test

You can see that 4 requests fall on v2 and 6 fall on V1. Theoretically, the more requests, the closer the number of requests fall on V2 is to 50% of the set weight

$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Copy the code

2. Header-based A/B testing

  • Change the marshalling file for version V2

Increased headernginx ingress. Kubernetes. IO/canary – by – the header: “v2”

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
    nginx.ingress.kubernetes.io/canary-by-header: "v2"
  labels:
    app: echoserverv2
  name: echoserverv2
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv2
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name:  echoserverv2
  namespace: echoserver
spec:
  selector:
    name:  echoserverv2
  type: ClusterIP ports: - name: echoserverv2 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv2 namespace: echoserver labels: name: echoserverv2 spec: template: metadata: Labels: name: echoserverv2 spec: containers: - image: mirrorgooglecontainers/echoserver: 1.10 name: echoserverv2 ports: - containerPort: 8080 name: echoserverv2Copy the code
  • Update access tests

The three hearder values whose header is v2:always v2:never v2:true are tested. It can be seen that when the hearder value is v2:always, all traffic flows to V2. When the hearder value is v2:never, all traffic flows to V1. The value is non-always /never. Traffic flows to services of the corresponding version based on the configured weight

$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Copy the code
  • The custom header – the value
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
    nginx.ingress.kubernetes.io/canary-by-header: "v2"
    nginx.ingress.kubernetes.io/canary-by-header-value: "true"
  labels:
    app: echoserverv2
  name: echoserverv2
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv2
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name:  echoserverv2
  namespace: echoserver
spec:
  selector:
    name:  echoserverv2
  type:  ClusterIP
  ports:
  - name:  echoserverv2
    port:  8080
    targetPort:  8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  echoserverv2
  namespace: echoserver
  labels:
    name:  echoserverv2
spec:
  template:
    metadata:
      labels:
        name:  echoserverv2
    spec:
      containers:
      - image:  Mirrorgooglecontainers/echoserver: 1.10
        name:  echoserverv2 
        ports:
        - containerPort:  8080
          name:  echoserverv2
Copy the code
  • Update the test

You can see that only when header is set to v2:never, the request traffic flows to the v2 version. The other traffic flows to the services of the unavailable version based on the weight Settings

$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn

$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Copy the code

3. Cookie-based flow control

Cookie is basically the same as header and is also an ingress automatic cookie value. If the cookie matches, traffic will flow to the matching back-end service

  • Updated the choreographer file of v2 version
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
    nginx.ingress.kubernetes.io/canary-by-header: "v2"
    nginx.ingress.kubernetes.io/canary-by-header-value: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_shanghai"
  labels:
    app: echoserverv2
  name: echoserverv2
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv2
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name:  echoserverv2
  namespace: echoserver
spec:
  selector:
    name:  echoserverv2
  type:  ClusterIP
  ports:
  - name:  echoserverv2
    port:  8080
    targetPort:  8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  echoserverv2
  namespace: echoserver
  labels:
    name:  echoserverv2
spec:
  template:
    metadata:
      labels:
        name:  echoserverv2
    spec:
      containers:
      - image:  Mirrorgooglecontainers/echoserver: 1.10
        name:  echoserverv2 
        ports:
        - containerPort:  8080
          name:  echoserverv2
Copy the code
  • Access to the test

You can see that the access effect is the same as the header, except that the cookie does not have a custom value

$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged

$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn

# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:01:52]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn

# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:02:25]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai=always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Copy the code

conclusion

Grayscale release can ensure the stability of the overall system. At the initial grayscale, the new version can be tested, found and adjusted to ensure its impact. The above content introduces ingress-Nginx actual Canary Annotation in detail through examples. Blue-green publishing and Canary publishing can be easily implemented with ingress-nginx

other

About blue-green releases, Canary releases, and A/B testing

  • Blue green release

In the blue-green deployment, there are two systems: one that is providing services and is marked green; The other is a system ready for release, marked “blue”. Both systems are fully functional and running systems, but the system version and external services are different. Initially, there was no system, no blue or green. Then, the first system is developed and goes online directly. There is only one system in this process, and there is no blue or green. Later, a new version is developed to replace the old version online, and a whole new system is built using the new version code in addition to the online system. At this time, two systems are running. The old system is green and the newly deployed system is blue. The blue system does not provide external services. What is it used for? Used for pre-release testing, any problems found during the test can be directly modified on the blue system, without interfering with the system users are using. (note that the two systems when coupled to absolutely guarantee no interference) after repeated testing, modification, blue system verification, determine to achieve on-line standard, directly to the user to switch to a blue system: switch after a period of time, is still blue and green two systems coexist, but the user access system is blue. During this period, observe the working status of the blue system (new system), and switch back to the green system if there is a problem. When the blue system that provides services to the outside is believed to work properly and the green system that does not provide services to the outside is no longer needed, the blue system officially becomes the new green system that provides services to the outside. The original green system can be destroyed, freeing up resources for the deployment of the next blue system. Blue-green deployment is just one of the go-online strategies, and it’s not a one-size-fits-all solution. Blue-green deployments can be implemented easily and quickly assuming that the target system is very cohesive. If the target system is quite complex, you need to think carefully about how to switch, whether the data between the two systems is needed, and how to synchronize.

  • Published by Canary

Canary release (Canary) is also a release strategy, and domestic often said grayscale release is the same kind of strategy. Blue-green deployment is to prepare two systems, switch between them, and the Canary strategy is to have only one system and gradually replace that system for example, the target system is a set of stateless Web servers, but there are a lot of them, let’s say 10,000 of them. At this point, blue-green deployment will not work, because you cannot request 10,000 servers dedicated to deploying a blue system (in the definition of blue-green deployment, the blue system should be able to handle all access). One possible approach is to prepare just a few servers on which to deploy the new version of the system and test and verify it. After the test passed, I was afraid to update all the servers immediately for fear of accidents. Start by updating 10 of the 10,000 servers online to the latest systems, and then observe and verify. After verifying that there are no exceptions, update all remaining servers. This method is canary publishing. More control can be exercised, such as setting a lower weight for the first 10 updated servers, controlling the number of requests sent to those 10 servers, and then gradually increasing the weight and number of requests. This control is called “traffic sharding” and can be used for both Canary releases and later A/B testing. Blue-green deployment and Canary publishing are two publishing strategies, and neither is a panacea. Sometimes you can use both, sometimes you can only use one or the other.

  • A/B testing

The first thing to be clear about is that A/B testing and blue-green deployment and canaries are two very different things. Blue-green deployment and Canary are release strategies that aim to ensure the stability of a new system, focusing on bugs and hidden dangers of the new system. A/B test is an effect test in which multiple versions of services are available at the same time. These services have been tested enough to meet the online standards, but there is no difference between new and old services (they may be deployed in blue-green mode when they are online). A/B testing focuses on the actual effects of different versions of the service, such as conversion rates, order levels, etc. In A/B testing, multiple versions of services are running online at the same time. These services usually have some differences in experience, such as page style, color, and operation flow. Analyze the actual effect of each version and select the version with the best effect. In A/B testing, you need to be able to control the distribution of traffic, for example, 10 percent for version A, 10 percent for version B, and 80 percent for version C.