About the author: Feng Yong (Hua name Lu Jing), a senior technical expert, holds a PhD degree in computer science from Northwestern Polytechnical University. He has more than 10 years of design and development experience in the fields of high-performance computing, big data and cloud computing, focusing on scheduling, resource and application management. He has also been deeply involved in the development and commercialization of related open source projects, such as OpenStack, Mesos, Swarm, Kubernetes, Spark, etc., and led related development teams at IBM.
preface
Anyone who hasn’t heard of Kubernetes in cloud computing is like someone who doesn’t know that hot pot in Chongqing must have hot peppers. Kubernetes has become the de facto standard platform for managing data centers, like Android on phones and Windows on laptops. Around Kubernetes, the open source community has built a rich technology ecology, whether it is CI/CD, monitoring operation and maintenance, or application framework, security and anti-intrusion, users can find their own projects and products. However, once the scenario is extended to a multi-cluster, hybrid cloud environment, the open source technologies that users can rely on are few and far between, and often not mature and comprehensive.
In order to make it as easy for developers and users to use their familiar open source projects and products to develop functionality in a multi-clustered and mixed environment as they would on a single Kubernetes cluster platform, OCM (Open Cluster Management), jointly launched by RedHat, Ant and Aliyun, aims to solve the problem of life cycle Management of resources, applications, configurations, policies and other objects in a multi-cluster, mixed environment. Currently, OCM has submitted an incubator application to CNCF TOC for a Sandbox-level project.
Project official website: https://open-cluster-management.io/
Development history of multi-cluster management
Going back a few years, when the industry was still concerned/debating whether Kubernetes was production-ready or not, there were some of the first players to sign on to the “multi-cluster federation” technology. Most of them are the pioneers of Kubernetes practice on the volume far beyond the average level, from the earliest Red Hat and Google entry to the Kubefed V1 attempt, and then to learn from the experience and launch Kubefed V2 together with IBM. In addition to these large enterprises in the production practice of Kuberentes scenarios to explore multi-cluster federated technology, in the commercial market, each vendor based on the Kubernetes packaging service products also experienced the process of evolution from single cluster product service to multi-cluster form, hybrid cloud scenario. In fact, both enterprises themselves and business users have common needs, focusing on the following aspects:
Multi-geographic issues: When a cluster needs to be deployed on a heterogeneous infrastructure or across a wider geographic area
The Kubernetes cluster relies on ETCD as the data persistence layer. ETCD, as a distributed system, has requirements on the network delay between members of the system and some restrictions on the number of members. Although it can be adapted by adjusting parameters such as heartbeat if the delay can be tolerated, However, it can not meet the global deployment requirements of transnational and trans-continental, nor can it guarantee the number of available areas in large-scale scenarios. Therefore, in order to make ETCD run stably at least, ETCD is generally planned into multiple clusters according to the region. In addition, hybrid cloud architectures are increasingly accepted by users on the premise of business availability and security. It is difficult to deploy a single ETCD cluster across cloud service providers, and accordingly, the Kubernetes cluster is split into multiple clusters. When the number of clusters gradually increases and administrators are overwhelmed, it is natural to need an aggregated management and control system to manage and coordinate multiple clusters simultaneously.
Scaling issues: When single cluster scalability hits a bottleneck
To be sure, the open source version of Kubernetes has an obvious scale bottleneck, but what’s worse is that it’s hard to really quantify the scale of Kubernetes. The Kuemark suite was initially provided by the community to verify the performance of the cluster, but the reality was that Kuemark was limited to scheduling Workload scaling over and over again with different number of nodes. However, in practice, the reasons for the performance bottleneck of Kubernetes are complex and there are many scenarios. Kuemark is difficult to describe the scale of multiple clusters comprehensively and objectively, so it can only be used as a reference scheme under very coarse-grained conditions. Community support for measuring cluster capacity in multiple dimensions in the form of a scale envelope was followed by a more advanced cluster test suite, Perf-Tests. As users become more aware of the scale issues, they can plan the distribution of several Kubernetes clusters in advance based on the actual scenarios (such as IDC size, network topology, etc.), and the need for multi-cluster federation emerges.
Disaster resilience/isolation issues: when more granular isolation and resilience requirements arise
Disaster resilience of business applications is realized by scheduling policies within the cluster and deploying the applications to different granularity infrastructure available areas. Combined with network routing, storage, access control and other technologies, it can solve the problem of service continuity after the failure of available area. But how do you resolve failures at the cluster level, or even the cluster management control platform itself?
ETCD as a distributed system can naturally solve the problem of most nodes failure, but unfortunately in practice ETCD service may still be down, may be the management of the operation error, may also be the occurrence of network partition. In order to avoid “destroying the world” when an ETCD goes wrong, a more granular disaster recovery strategy is often provided by reducing the “explosion radius”. For example, in practice, it is more inclined to build multiple clusters within a single data center to avoid brain-splitting problem, and at the same time, make each cluster an independent autonomous system, which can run completely even in the case of network partition or offline control of higher level, and at least keep the site stable. There is a natural need to manage multiple Kubernetes clusters simultaneously.
On the other hand, the isolation requirement also comes from the lack of multi-tenancy capability of the cluster, so the isolation strategy at the cluster level is adopted directly. The good news is that Kubernetes’ control surface fairness/multi-tenant isolation is being built brick by brick. Through the APIPriorityAndFairness feature, which entered Beta in version 1.20, it is possible to actively customize the traffic soft isolation strategy based on the scenario. Instead of passively limiting traffic through a penalty like an ACL. If the cluster is divided into multiple clusters at the beginning of the cluster planning, then the problem of isolation will be solved. For example, we can assign exclusive clusters for big data based on business, or exclusive clusters for specific business applications, etc.
The main functionality and architecture of OCM
OCM is designed to simplify the management of multiple Kubernetes clusters deployed in a mixed environment. Can be used to extend multi-cluster management capabilities for different management tools in the Kubernetes ecosystem. OCM summarizes the basic concepts required for multi-cluster management and argues that in multi-cluster management, any management tool needs to have the following capabilities:
1. Understand the definition of cluster;
2. Select one or more clusters by some scheduling method;
3. Distribute configuration or workload to one or more clusters;
4. Govern the access control of users to the cluster;
5. Deploy management probes to multiple clusters.
OCM uses a Hub-Agent architecture, which includes several multi-cluster management primitives and basic components to achieve the above requirements:
● ManagedCluster API to define the ManagedCluster, and OCM will install agent named Klusterlet in each cluster to complete cluster registration, life cycle management and other functions.
● Use the Placement API to define how the configuration or workload is scheduled to which clusters. Scheduling results are stored in the PlacementDecision API. Other configuration management and application deployment tools can use the placementDecisiono to determine which clusters need to be configured and which applications need to be deployed.
● Define the configuration and resource information distributed to a cluster through the ManifestWork API.
● Cluster is grouped through the ManagedClusterSet API and provides boundaries for user access to the cluster.
● Define how the management probe will be deployed in multiple clusters and how it will communicate securely and reliably with the control surface on the hub side through the ManagedClusteraddon API.
The architecture is shown in the figure below, where Registration is responsible for cluster registration, cluster lifecycle management, management plug-in registration and lifecycle management; Work is responsible for resource distribution; Placement is responsible for scheduling cluster loads. On top of this, developers or SRE teams can easily develop and deploy management tools in different scenarios based on API primitives provided by OCM.
By leveraging OCM’s API primitives, you can simplify the deployment and operation of many other open source multi-cluster management projects, as well as extend the multi-cluster management capabilities of many Kubernetes’ single cluster management tools. Such as:
1. Simplify the management of multi-cluster network solutions such as Submariner. Use OCM’s plug-in management capabilities to centralize the deployment and configuration of Submariner into a unified management platform.
2. Provide rich multi-cluster responsible scheduling strategy and reliable resource distribution engine for application deployment tools (Kubevela, ArgoCD, etc.).
3. Expand the existing Kuberenetes single cluster security Policy governance tool (Open Policy Agent, Falco, etc.) to make it have the capability of multi-cluster security Policy governance.
OCM also has two management plug-ins built in for application deployment and security policy management. The application deployment plug-in adopts subscriber mode, which can obtain the resource information of application deployment from different sources by defining subscription channels. Its architecture is shown in the figure below:
In order to integrate closely with the Kubernetes ecosystem, OCM has implemented several designs of the Kubernetes SIG-Multicluster, including KEP-2149 Cluster ID: https://github.com/Kubernetes/enhancements/tree/master/keps/sig-multicluster/2149-clusterid
And the concept of clusterset in the KEP-1645 Multi-Cluster Services API: https://github.com/Kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api
Also in and other developers in the community to promote the Work of APIhttps://github.com/Kubernetes-sigs/work-api development.
Key advantages of OCM
Highly modular – freely selectable/tailored modules
The overall OCM architecture is very much like a “microkernel” operating system, where the OCM chassis provides services such as clustering metadata abstraction for core capabilities, while other extended capabilities are deployed as separate components that can be split apart. As shown in the above in the whole scheme of OCM in addition to the ability of the core part, the ability of other top can be tailored according to the actual demand, for example, if we don’t need complex cluster topology relationship, you can cut off the cluster grouping related module, if we don’t need any OCM distributed resources used only as metadata, You can even crop out the Agent component of the entire resource. This is also a good way to get users to log in to OCM, where they may need only a small amount of functionality in the beginning, then slowly introduce more feature components as the scenario expands, and even support hot plug on the running control surface.
More inclusive – the Swiss Army knife of complex use scenarios
At the beginning of the design, the whole OCM scheme takes into consideration the construction of advanced capabilities for some complex scenarios through the integration of some third-party mainstream technical solutions. For example, to support more complex rendering of application resources, OCM supports installing applications as Helm Chart and loading remote Chart repositories. The Addon framework is also provided to allow users to customize their needs with extensible interfaces, such as Submarine’s multi-cluster network trust solution based on the Addon framework.
Ease of use – Reduce complexity
In order to reduce user complexity and ease of migration to the OCM solution, OCM provides a traditional directive multi-cluster federated control process. It should be noted that the following mentioned features are still in development and will be available in future versions:
● Through ManagedClusterAction, we can issue atomic instructions to the managed clusters one by one, which is also the most intuitive way to arrange each cluster automatically as a central control system. A Management ClusterAction can have its own type of instruction, the content of the instruction, and the specific state of the instruction execution.
● With ManagedClusterView, we can proactively “project” the resources in the nanometer cluster to the multi-cluster federated backbone system. By reading the “projection” of these resources in the backbone, we can make more dynamic and accurate decisions in the federated system.
OCM practice in ant group
OCM technology has been applied to the Ant Cluster infrastructure as a first step to deploy OCM Klusterlets into managed clusters one by one using some kind of operational approach similar to the Community Cluster API. Thus, the meta-information of dozens of online and offline clusters in the ant domain is integrated into OCM. These OCM Klusterlets provide the underlying capability of multi-cluster management and operation for the upper layer of the product platform for future functionality expansion. Specifically speaking, the first step of OCM includes the following aspects:
Certifierless: In a traditional multi-cluster federated system, it is often necessary to configure the cluster access certificate for each cluster’s metadata, which is a required field in the cluster metadata model of Kubefed V2. Because OCM adopts the Pull architecture as a whole, there is no central active access to the actual cluster process when agents deployed in each cluster Pull tasks from the center, so the metadata of each cluster is just a placeholder for complete desensitization. At the same time, because the certificate information does not need to be stored, there is no risk of the certificate being copied and misappropriated in the OCM scheme
● Automated cluster registration: The previous cluster registration process with more manual operations lengths the collaboration and communication time while losing the flexibility to change, such as site level or machine room level flexibility. Manual verification is necessary in many scenarios. We can take full advantage of the audit and verification capabilities provided by OCM cluster registration and integrate them into the approval process tools within the domain to automate the entire cluster registration process and achieve the following goals:
(1) Simplify the cluster initialization/takeover process;
(2) More clearly control the authority of the management and control center.
● Automated cluster resource installation/uninstallation: Takeover mainly involves two things: (a) the application resources needed to install the management platform in the cluster and (b) the entry of cluster metadata into the management platform. For (a) resources that can be further classified into Cluster level and Namespace level, while (b) is generally a critical operation for the upper-level control system. The product is considered to have taken over the Cluster after the moment of metadata entry. Prior to the introduction of OCM, all the preparatory work had to be done manually and step by step. The entire process can be automated through OCM, simplifying the cost of human collaboration and communication. The essence of this is to streamline the cluster management as a process operation, and to define the concept of state on top of the cluster metadata so that the product backbone can streamline and automate the “chores” it needs to do. Once the cluster is registered in the OCM, the process of installing and uninstalling resources is clearly defined.
Through the above work, dozens of colonies in the ant domain are within the management scope of OCM. In the “Double 11” promotion, automatic creation and deletion of clusters have also achieved automatic access and deletion. In the future, we also plan to integrate with Kubevela and other application management technologies, and cooperate with them to complete the cloud primitive biological management capabilities in the ant domain, such as application and security policy.
OCM’s practice in Aliyun
At Aliyun, the OCM project is Kubevela https://github.com/oam-dev/kubevela
One of the core dependencies of undifferentiated application delivery for a mixed environment. Kubevela is a “one-stop” application management and delivery platform based on the Open Application Model (OAM), and is currently the only cloud-based native application platform project hosted by the CNCF Foundation. In terms of functions, Kubevela can provide developers with end-to-end application delivery model, as well as multiple multi-cluster-oriented operation and maintenance capabilities such as grayscale publishing, elastic scaling and observability, and can deliver and manage applications in a mixed environment with a unified workflow. Throughout the process, OCM was Kubevela’s main technique for implementing Kubernetes cluster registration, management, and distribution policies.
On the public cloud, the above characteristics of Kubevela combined with the multi-cluster nano management capability of Ali Cloud ACK can provide users with a powerful application delivery control plane, which can be easily realized:
● Mixed environment one key construction station. For example, a typical hybrid environment could be a public cloud ACK cluster (production cluster) plus a local Kubernetes cluster (test cluster) managed by ACK multiple clusters. In both environments, the provider of application components is often different, such as the database component might be MySQL in the test cluster, but AliCloud RDS product in the public cloud. In such a mixed environment, traditional application deployment and operation are extremely complex. Kubevela, on the other hand, makes it easy for users to define the artifacts to be deployed, the delivery workflow, and the differentiated configuration of different environments in a single deployment plan. This not only eliminates the tedious manual configuration process, but also greatly reduces the risk of release and operation with the power of automation and certainty of Kubernetes.
● Multi-cluster micro-service application delivery: the micro-service application under the cloud native architecture is often composed of diversified components, such as container components, HELM components, middleware components, cloud service components, etc. Kubevela provides users with a multi-component application delivery model oriented to microservice architecture. By using the distribution strategy provided by OCM, unified application delivery can be made in a multi-cluster and mixed environment, which greatly reduces the difficulty of running and managing microservice applications.
In the future, the AliCloud team, together with partners such as RedHat/OCM community, Oracle, Microsoft and others, will further improve Kubevela’s application layout, delivery and operation and maintenance capabilities for mixed environments, so that the delivery and management of micro-service applications in the cloud native era can be really “fast and good”.
Join a community
The OCM community is still in the early stages of rapid development, and interested businesses, organizations, schools, and individuals are most welcome to participate. Here, you can learn, build and promote OCM with technical experts from Ant Group, RedHat, and AliCloud, as well as Kubernetes’ core Contributor partners.
●GitHub: https://github.com/open-cluster-management-io
Low by understanding the OCM video: https://www.youtube.com/channel/UC7xxOh2jBM5Jfwt3fsBzOZw
To community weeks will meet all of you: https://docs.google.com/document/d/1CPXPOEybBwFbJx9F03QytSzsFQImQxeEtm8UjhqYPNg
● Free communication on the Kubernetes Slack channel # open-cluster-mgmt: https://slack.k8s.io/
Low to join browse key discussion group email: https://groups.google.com/g/open-cluster-management
● Visit the community website for more information: https://open-cluster-management.io/
This year, an Inclusion · Bund event will be held on September 10 as scheduled. As a global fintech event, it will continue to maintain the original intention of making technology more inclusive. On the afternoon of the 11th, major developers from the OCM community will present the best practices of multi-cluster and hybrid cloud architecture around OCM. You are welcome to join us offline and have face-to-face communication. Thank you for your attention and participation in OCM. Welcome to share with more friends who have the same needs. Let’s contribute to the further use experience of multi-cluster and hybrid cloud together!
Read this week
- RFC8998+ Babassl — Let the secret country sail to the far sea of stars
- MOSN sub-project Layotto: Open a new chapter of service grid + application runtime
- A new chapter of Cloud Native MOSN – Fusing Envoy and Golang Eco
- MOSN multi-protocol extension development practice
For more articles, please scan the code to pay attention to the official account of “Financial-grade Distributed Architecture”
