The author

Xu Di, Tencent cloud container technology expert. Ru Yingzhe, senior product manager of Tencent Cloud.

Abstract

In multi-cluster application distribution, the following differentiation problems are often encountered, such as:

  1. Label all distributed resources with uniform labels, such asapps.my.company/deployed-by: my-platform;
  2. Tag cluster information on resources distributed to subsets, such asapps.my.company/running-in: cluster-01;
  3. Adjust the number of copies applied in each cluster, the mirror name, and so on, such as having a name calledmy-nginx(The number of copies of the declaration is 3)DeploymentThe application needs to be distributed to cluster-01, cluster-02, and cluster-03. I want the number of replicas in these three clusters to be 3, 5, and 7 respectively.
  4. Before distributing the application to cluster cluster-01, adjust the configuration of the application in the cluster, such as injecting a Sidecar container.
  5. Encounter some special scenarios, such as big promotion, dynamic expansion, gray upgrade application, hope to be able to operate for a cluster, the change scope is small, does not affect other clusters, at the same time when the problem occurs, can be rolled back in time, restore to the state before the change;
  6. If multiple differentiated configurations are defined, how to resolve conflicts among them?

Brief introduction to the Source Clusternet Project

Clusternet (Cluster Internet) is a cloud native control project of Tencent cloud open source, which combines multi-cluster management and cross-cluster application choreography. It makes using multi-cluster as easy as surfing the Internet. Regardless of whether your Kubernetes cluster is running on public, private, hybrid or edge clouds, you will have a consistent management/access experience to centrally deploy and coordinate multiple clusters of applications and services using the K8s API.

Clusternet uses the Addon plug-in for easy installation, maintenance and integration, making it easy to manage millions of Kubernetes clusters, making cloud computing as ubiquitous as the Internet, free and convenient.

Clusternet support to different cluster distribution and management of various applications including native Kubernetes all kinds of resources (Deployment/StatefulSet/ConfigMap/Secret, etc.), all kinds of CRD resources, And the HelmChart app and so on.

How does Clusternet address these differentiated configuration challenges

When designing the application distribution model, Clusternet takes these scenarios into account. It does not want to introduce too many complex designs, and minimizes user definitions. It is streamlined, easy to configure, scalable, and easy to roll back changes.

If we summarize the above differentiation problems, they can be roughly classified into the following two categories:

  1. Universal or global configuration, such as undifferentiated labeling and pre-configuration for certain resources.

  2. Cluster-specific configurations, such as changing the number of copies of Deployment in a cluster, upgrading mirrors, adding Sidecar containers, etc.

The following figure shows Clusternet’s multi-cluster application distribution model, in which the green modules need to be created by users, and the purple modules are resource objects for circulation within Clusternet. Clusternet provides the Kubectl plug-in, which allows you to create resources by using the “Kubectl Clusternet apply” command. Welcome to Clusternet – The Next Generation Open Source Multi-Cluster Management and Application Governance Project to learn about the concepts in the diagram.

Clusternet resource distribution model USES the design of loosely coupled, users do not need to change or to rewrite the existing resource objects, only need to define additional distribution strategy (Subscription) and differential configurations (Localization/Globalization) can realize the application of cluster distribution.

Localization and Globalization

In Clusternet, each registered cluster has its own namespace. Therefore, we define two CRDS, Localization and Globalization, respectively to declare differentiated configuration. Where Localization describes the differential configuration policy of namespace-scoped, which can be used to configure a single cluster, such as the number of copies of Deployment in the cluster, etc. Globalization describes cluster-scoped differentiated configuration policies, such as modifying the generic configuration of a HelmChart.

Override strategy

Clusternet also provides two Overide policies: ApplyLater (the default policy) and ApplyNow. ApplyLater means the differential configuration of the Localization/Globalization does not apply to the resource, Will only in the newly created object of Description or HelmChart/Subscription/Description updates take effect when each resources such as object. ApplyNow means that it takes effect immediately after it is created. Clusternet applies the differentiated configuration to all matched objects and delivers it to the corresponding subset immediately.

Priority Priority

Both support management and configuration based on Priority. The value of Priority ranges from 0 to 1000. A smaller value indicates a lower Priority, and the default value is 500. When you do a differentiated rendering, Clusternet is organized in the order of Globalization -> high-priority Globalization -> Localization -> Localization. Apply the declared Override in turn.

Thanks to this two-stage priority-based differentiated configuration capability, Clusternet can easily support multi-cluster scenarios such as blue-green release, Canary release, and version upgrade. You can define multiple Globalization and Localization objects and set different priority policies during Localization.

Patch operation

Clusternet supports Override in two formats, JSON Patch (RFC 6902) and JSON Merge Patch (RFC 7396). To compare JSON patch with JSON Merge Patch, you can view JSON patch and JSON Merge Patch, or refer to the following typical example.

A typical example

Let’s look at a few typical differential configuration scenarios. In the following example, we unified display through Localization object. The Spec definition is the same for both. The only difference is the scope and priority of the two. When you actually use it, you can rewrite it as needed.

Add/update labels

If we want to add or update tags to an object, we can define the following Localization object. Replace the metadata.namespace value with the namespace of the registered cluster.

apiVersion: apps.clusternet.io/v1alpha1 kind: Localization metadata: name: nginx-local-overrides-demo-label namespace: Clusternet-5l82l # Please update this value to the namespace of the corresponding cluster spec: overridePolicy: The value ranges from 0 to 1000, with a smaller value indicating a lower priority. The default value is 500. Apps /v1 kind: Deployment name: my-nginx namespace: foo overrides: # MergePatch # Specifies the type # value of Override in either YAML or JSON format. Value: '{"metadata":{"labels":{"deployed in-cluster":" ClusterNET-5L82L "}}}'Copy the code

Multiple overrides can be defined in a Localization object. In the above example, we only define override named Add-update-labels, whose value is a string in JSON format. The objective is to add or update the tag deployed In-Cluster: ClusterNET-5L82L to the object defined by spec.feed.

Here the override value can also be in YAML format, as shown in the following example.

apiVersion: apps.clusternet.io/v1alpha1 kind: Localization metadata: name: nginx-local-overrides-demo-label namespace: Clusternet-5l82l # Please update this value to the namespace of the corresponding cluster spec: overridePolicy: The value ranges from 0 to 1000, with a smaller value indicating a lower priority. The default value is 500. Apps /v1 kind: Deployment name: my-nginx namespace: foo overrides: # override value - name: override value - name: Add-update-labels type: MergePatch # value Can be in YAML format or JSON format. # below are examples of the yaml format value: | - metadata: labels: deployed - in - cluster: l82l clusternet - 5Copy the code

Number of mirrors and copies to be replaced

Override can also be specified as JSONPatch. In actual use, you can select an appropriate override type as required.

Using the following example, you can change the number of copies of Deployment foo/my-nginx in the Clusternet-5l82L subset to 3, replace the container image to nginx:1.14.0-alpine, and add a new annotation foo: bar.

apiVersion: apps.clusternet.io/v1alpha1 kind: Localization metadata: name: Nginx-local-overrides -demo-image-replicas namespace: clusternet-5l82l The value ranges from 0 to 1000, with a lower priority. The default value is 500. Priority: 400 feed: # Apps /v1 kind: Deployment name: my-nginx namespace: foo overrides: # override value - name: override value - name: Annotations Type: JSONPatch # value The scale-and- add-Annotations type: JSONPatch # value can be a YAML format or a JSON format. value: |- - path: /spec/replicas value: 3 op: replace - path: "/spec/template/spec/containers/0/image" value: "Nginx :1.14.0-alpine" op: replace-path: /metadata/ Annotations Value: foo: bar op: addCopy the code

Inject the Sidecar container

We can also inject the Sidecar container for Deployment Foo /my-nginx instances in clusternet-5L82L subgroup by Localization, as shown in the following example:

apiVersion: apps.clusternet.io/v1alpha1 kind: Localization metadata: name: Nginx-local-overrides -demo-sidecar Namespace: clusternet-5l82l The value ranges from 0 to 1000, with a smaller value indicating a lower priority. The default value is 500. Apps /v1 kind: Deployment name: my-nginx namespace: foo overrides: # override value - name: override value - name: Inject -new-container type: JSONPatch # value The value can be in YAML format or JSON format. Value: | - op: add path: "/ spec/template/spec/containers / 1" value: name: "redis - container" image: "redis: 6.2.5"Copy the code

Through Localization and Globalization, we can not only make the above differential configuration, but also have more scenes to be discovered.

For a hands-on experience, Clusternet provides examples for you to follow the README steps to practice multi-cluster application distribution.

Join us

For the open source Clusternet project, please visit github.com/clusternet/… Like support, welcome to join us to contribute more features.

A link to the

[1] github.com/clusternet/…

[2] github.com/clusternet/…

[3] krew.sigs.k8s.io/plugins/

About us

More about cloud native cases and knowledge, can pay attention to the same name [Tencent cloud native] public account ~

Benefits:

Background reply [manual] (1) the public, can obtain "tencent cloud native roadmap manual" & "tencent cloud native best practices" ~ (2) public reply background 】 【 series, can obtain the series of 15 + 100 super practical cloud native original dry anthology, contains Kubernetes efficiency of authors and K8s performance optimization practice, best practices, etc.Copy the code