Summary: This article will first cover the challenges of cloud native application delivery and management, then the KubeVela and OCM technology principles behind it, and finally the overall best practices, as well as a full Demo.
Authors: Feng Yong, Sun Jianbo
Hello everyone, I am very happy to be here at KubeCon China Summit to share with you. Today’s theme is “Building and Managing Multi-cluster Applications with A Consistent Experience”, featuring KubeVela and OCM, both CNCF open source projects. The presentation is divided into three parts, starting with an introduction to the challenges of cloud native application delivery and management, followed by an introduction to KubeVela and OCM technology principles behind this, followed by an overview of best practices, and a full Demo.
background
With the flourishing of the cloud native ecosystem, Kubernetes has become the standard integrated interface for infrastructure, and more and more infrastructure capabilities have become declarative apis out of the box. The popularity of CRD Operator**[1]** has also made operational and maintenance capabilities more declarative and automated. As shown in Figure 1, there are now thousands of projects in CNCF Ecology **[2]**, ranging from low-level infrastructure to upper-level application development.
But the abundance of choice is a gift and an annoyance to r&d engineers. Different application architectures involve different development languages, technical frameworks, packaging methods, product forms, cloud resources, and operations. In the software life cycle, there are also great inconsistencies in the environment corresponding to development, testing, pre-release, production deployment and application delivery deployment experience. Switching to the cloud native stack involves a lot of complex new knowledge to learn, which is like writing to a large number of operating system underlying apis without a programming framework and standard library, making application development less efficient.
How to make good use of the flourishing technology ecosystem of cloud native, and enable business r&d personnel to obtain consistent low threshold experience, while delivering applications safely and reliably is a huge challenge facing the industry.
Industry typical practices and challenges
To address this last mile of application delivery, industry practices typically fall into two broad categories.
The first type is to build an internal PaaS platform based on Kubernetes for its own business scenarios, hide the Interface of Kubernetes platform and provide its own platform API. This model is usually found in larger companies and needs to be supported by teams that are proficient in Kubernetes and the business. However, as time goes by, business scenarios become complex and demands gradually increase, internal PaaS will face the difficulties of insufficient expansion capacity and difficult maintenance, and finally have to overturn the dilemma of redo. This problem is particularly obvious in the practice of Alibaba’s early application of cloud bio-transformation. On the other hand, large companies usually have different business teams. Due to the lack of top-level design, the PaaS platforms built by different teams can easily become independent chimneys. A large number of similar capabilities can only be built repeatedly and cannot be reused, let alone managed uniformly. The different platforms for each different scenario impose new concepts and learning burdens on the higher level business team.
In view of the problems of the first type of scenario, the industry is gradually inclined to the container platform layer to provide the K8s native API, which is responsible for providing stable cloud native ecological capabilities, while not affecting flexibility and scalability. Application delivery extends the second approach by packaging applications (such as Helm Chart, etc.) and deploying them directly to a container platform through a Jenkins/Gitlab Pipeline. Based on the pattern of the traditional CI ecosystem tools directly connecting to the container platform, as shown in Figure 2, the core of this approach is to simplify the usage threshold by building an abstract system through scripts, configurations, etc. This approach is currently a popular solution in the industry, with a clear division of benefits. The container platform layer serves as the Infra/SRE team to maintain the infrastructure capacity, while the application delivery takes full advantage of the existing CI system within the enterprise, without the need to build a platform.
This approach solves the problem of application management to a certain extent. Even small and medium-sized enterprises lack the ability to maintain container platforms, they can purchase cloud services to solve the problem. Meanwhile, it can provide one-stop consistent experience for business developers. But the main problem lies in the application delivery front, and enterprises of all sizes and scenarios will quickly encounter these challenges:
- ** lacks automation and requires a lot of manual maintenance work. For example, due to sudden network problems or a problem with the Pipeline system itself, all releases are interrupted and manual intervention is required, with no self-healing capability.
- ** lacks a unified application model. ** Unity of the application model is extremely important, both as a complete deliverable of the application and as a core single source of truth for the application. The lack of a unified model description as the application delivery entrance will lead to different people can change the system from multiple places, such as through CI Pipeline system, or directly change Kubernetes, over time, the system configuration will appear very inconsistent, resulting in failure.
- Security risks exist. Application delivery based on CI Pipeline system construction, the security domain of CI/CD system usually does not have isolation, that is, CI/CD is completed through a system, and the secret keys of all clusters of infrastructure are stored in the same system. Once breached by hackers, it is easy to cause very large system security risks. For example, the code coverage statistics tool Codecov suffered a security incident in April this year [3], and all CI keys configured by the project using the product were leaked. On the other hand, more and more open source software is adopted into enterprise production systems, and the integration of these software also increases security risks.
- ** High maintenance costs. * * through the script and simplify template developers mind is the core of this model, but the script and the maintenance of the template itself if the lack of open standards and ecological, will soon be lack of energy, gradually becomes no longer nobody dare to touch the mysterious code, extremely dependent on the system’s initial builders experience, difficult to continue and iteration.
Therefore, how to build a stable and reliable application delivery and management system has become the key we need to explore.
How to build stable and reliable application delivery and management system?
How to ensure stable, reliable, and secure application delivery, we’ll look at the solutions separately. First, to address the problem of stability and reliability for large-scale application delivery, we took a cue from Kubernetes’ design.
Inspiration from Kubernetes
Kubernetes has two core concepts, one is declarative API and the other is end-state oriented control loop.
Declarative is the best expression of user-side abstraction, which greatly simplifies the user’s mind from describing the process to describing the result. It’s easy to see why declarative thinking can simplify the mind. Take the example of eating. The traditional process-based description is like you’re eating at home. Declarative description is when you go to a restaurant, you tell the waiter what you want to order, and eventually the restaurant will bring it to you.
On the other hand, the end-state control loop is also the best way to ensure reliability. This design principle requires our system to have the following characteristics:
- Automation, the ability to always run towards the final state and continue running if the state is not reached. The characteristics of automation also ensure that our system will not be interrupted because of a little instability, with strong self-healing ability.
- Convergence, that is, the result can be close to the final state during the operation of the system, and the final result can be constantly approached with each execution.
- Idempotent, which means that we have to execute the same process multiple times to make sure that the results are consistent, that there are no unexpected problems just because we execute it multiple times.
- Deterministic means that as long as the system runs normally, it can continue to execute until a certain result is reached.
These four features are also central to large-scale application delivery.
Application delivery security principles
Then we look at security. The core is simply to treat the security of the application delivery system as if it were a production environment. The CNCF Foundation also published application Delivery Best Practices **[4] and a White Paper [5]** in April 2021, which mentioned some security principles for application delivery.
- Delivery environment isolation, on the one hand, means that different delivery destinations should be isolated, delivery systems should not have direct access to all Kubernetes clusters, and different environments should also be isolated from each other. On the other hand, security domains should also be isolated for different phases of the CI/CD application life cycle, using separate secret key information.
- Integration checks, the principle that we do necessary security checks on the integrated open source tools, the dependencies we use, and the application’s own mirrors during delivery.
- Minimum permissions, this principle is easier to understand, specific to practice is to do the split permissions. Such as using tokens instead of permanent keys, adding necessary approval processes, and using key management tools. In particular, when using cloud resources, it is necessary to split the permissions of the secret key, such as using a sub-account mechanism, to avoid a single point of failure caused by a secret key leak and failure to recycle it in time.
- Audit and security monitoring, this principle requires that our application delivery and management system itself should also have good observability, with delivery behavior audit and overall security monitoring.
Core elements of end-state oriented application delivery system
To sum up, a stable, reliable and secure application delivery system requires core elements as shown in Figure 4 below.
The entry point of the system is a complete application model that covers different deliverables, cloud resources, and various configurations of environment orchestration. It is the unified entry point and the only basis for application delivery.
At the same time, this application model adopts the declarative way to face business users, provides the business layer with the ability of abstraction for the use scenario, can reduce the user’s threshold of use, reduce the complexity of the underlying API, and has the ability of flexible expansion. The delivery system interacts with the declarative API by pulling. The user only needs to describe the final state in the model layer, and the final delivery system converges toward this final state.
Within the delivery system, there are two core functions: choreography and deployment. At the same time, behind these two core functions, there should always be a continuous end-state control cycle, which is the cornerstone of automation and certainty, as well as the ability to monitor and audit deliverables and delivery processes.
Choreography addresses the direct dependencies of different deliverables, deployment sequencing, data transfer, and multi-cluster, multi-environment configurations, but the complexity of choreography should not be exposed to users. The orchestration execution engine of the delivery system should be able to automate orchestration based on a declarative end-state described by the user, rather than having the user write the orchestration process manually. Deployment is to solve the deployment of different delivery, such as cloud resources, Kubernetes resources, as well as the delivery problem of multiple clusters, need to be able to provide sufficient scalability, ensure that different deliverables can be deployed, at the same time can complete deployment in the security conditions of multi-cluster environment isolation.
Next generation cloud native application delivery and management
KubeVela is the most modern application delivery platform built with this philosophy in mind, making your application delivery and management easier, easier, and more reliable in today’s popular hybrid, multi-cloud environment. ** Evolution has hundreds of contributors participating in code contributions, absorbing engineering practices from a series of companies in different fields such as Ali, Ant, Tencent, Byte, the fourth Paradigm, Gitlab, and making full use of the ecological dividend of a hundred flowers blooming in the cloud native field to achieve application delivery and management.
Standard, Open application Model (OAM)
There is a complete Application Model behind KubeVela, that is OAM (Open Application Model). OAM is an application model jointly released by Alibaba and Microsoft in 2019. Now, it has been applied into cloud products by alibaba, Microsoft, Oracle, Salesforce, Tencent and many other cloud vendors. Meanwhile, IT has also been issued as the industry standard of “Cloud Computing Open Application Architecture” by The National Information and Communication Academy.
As shown in Figure 6, THE OAM model abstracts complex application delivery and management into four core concepts: application, component, operation and maintenance capability, and workflow, enabling users to describe the entire content of application delivery and management through a single configuration file. OAM has sufficient scalability to meet the needs of applications delivered to different clouds and environments. At the same time, OAM is not bound to the Kubernetes ecosystem. If out-of-the-box KubeVela does not meet the needs, the OAM community also welcomes users to participate in the construction of the model layer, according to the model independent implementation.
Application delivery control plane based on Kubernetes
KubeVela is a microkernel architecture, its core is a Kubernetes-based application delivery and management control plane, with self-healing, idempotent, convergence and determination of these four characteristics. Minimal deployment KubeVela requires only 0.5 cores and 200 MB of ram to deliver hundreds of applications. KubeVela also has a series of plug-ins out of the box, including multi-cluster delivery, cloud resource management, GitOps, Helm Chart, observability, and even KubeVela’s OWN UI console is integrated through plug-ins.
KubeVela is also not limited to the Kubernetes ecosystem. The official built-in cloud resources plug-in can integrate any Terraform module, deliver different cloud resources, and even virtual machines. Thanks to the OAM model and the extensibility principle that KubeVela has strictly followed since its design, users can easily add and extend the capabilities of the ecosystem.
Secure and Reliable Multi-cluster Management Technology (OCM)
The multi-cluster technology behind KubeVela is also at the heart of ensuring stable, secure, and large-scale delivery of applications. As we know, multi-cluster has long been a common form of internal infrastructure in enterprises due to geographical, scale, and disaster recovery considerations. However, Kubernetes’ native management capabilities remain at the single-cluster level for now. Each cluster can operate stably and autonomously, but it lacks the overall management capability across multiple clusters. To form a stable and unified application delivery and management platform, the following capabilities are required:
- O&m managers can obtain information such as water level changes and node health status of multiple clusters
- Can a business owner decide how to deploy application services in different clusters
- Application o&M personnel can obtain service status and issue relocation policies
Thanks to the Open Cluster Management (OCM) multi-cluster technology jointly initiated by RedHat, ant and ali cloud, KubeVela can easily manage the life cycle of objects such as resources, applications, configurations, and policies in multi-cluster and mixed environments. OCM enables users to easily deliver and manage applications using familiar open source projects and products in a multi-cluster and hybrid environment as well as on a single Kubernetes cluster platform.
Compared with existing multi-cluster technologies, OCM architecture has technological advantages in the following aspects to meet users’ core demands for secure, stable and large-scale delivery of cloud native applications:
- ** Modular: **OCM management platform consists of management primitives, base components, and many extensible components that can be freely combined like Lego bricks to build a set of functionality to meet user needs. This concept is also a natural fit with KubeVela, which together provides users with full ecological scalability.
- ** Scale: **OCM’s infrastructure inherits Kubernetes’ open, extensible features. Just as a single Kubernetes cluster can support thousands of nodes, OCM is designed to support thousands of managed clusters.
- ** Security: **OCM is designed on the basis of zero trust and minimal permissions. Registration communication between the management component and the managed agent adopts a two-handshake mode. In addition, the update of certificates, the management of access rights and the distribution of access tokens also adopt similar design with Kubernetes, which is realized by the corresponding extensible components through Kubernetes native API.
- ** Extensibility: **OCM provides a programming framework and management API for extensible components. Together with basic components and management primitives, it is easy to migrate projects from Kubernetes ecosystem to OCM multi-cluster management platform. Through the programming framework, the OCM community has implemented many Kubernetes’ management capabilities in a multi-cluster environment. Thanks to this, KubeVela and OCM can easily achieve deep integration.
Secure and consistent application delivery experience with KubeVela and OCM
KubeVela and OCM are inherently complementary. KubeVela is responsible for delivery and life cycle management of the application layer, while OCM manages the life cycle of cluster infrastructure resources. Working closely together, they provide end-to-end capabilities for application delivery and lifecycle management in a multi-cluster environment.
As shown in Figure 8, with KubeVela and OCM, the user’s delivery process in different environments is fully automated, saving a lot of manual work.
- The same application, business developers, as long as describing a component can bind different operational capacity in different delivery environment run independently, such as the development environment can be used to end the cloud alignment, and advance production environment can bind observability strategy, etc., without having to repeat such as mirror, such as port, environment variables, the parameters of the component level, greatly save the mental burden.
- On the other hand, in the control plane, the deployment of different applications across different environments can be automated in a single delivery process, with multiple approval options or automatic execution.
More importantly, based on KubeVela and OCM, secure GitOps multi-cluster application delivery can be built entirely based on a subscription model.
As shown in Figure 7, the CI flow is still the same as before, but the CD side describes a separate configuration repository. At the Git repository configuration level, the permissions of the two repositories are completely isolated. Through the unified application model, users can describe the complete application in the configuration repository, and KubeVela can also monitor the change of the complete application description, so as to actively pull the application configuration. During this process, the Git repository does not hold any secret keys to the delivery system. After the orchestration of the delivery system is completed, the control data is delivered through OCM, and the whole process is actively pulled by the runtime cluster. The delivery system does not have the secret key of any subset group centrally controlled. After the control data is delivered to the subset, the GitOps agent of the subset can take the initiative to acquire the changes of the deliverables and form a new autonomy.
We can see that each link works independently and has its own system. Each link is authorized and audited according to the need, so it is safe and reliable. The collaboration between KubeVela and OCM enables unified orcheography and management of hybrid cloud, multi-cluster, and edge applications, ultimately enabling secure delivery of cloud-side collaboration.
Start your end-to-end platform journey
As we look at the design principles of the application delivery and management engine with KubeVela and OCM at its core, I can see that the biggest benefits of this model are automation, low barriers, and security. So how do you unfold the practice? Starting with the latest Version of KubeVela V1.2, we have enhanced the complete end-to-end user practice so that you can visually deliver and manage applications.
First, KubeVela’s architecture is built entirely on microkernels and a highly extensible model that allows users to build their own cloud-native solutions on demand at minimal cost. KubeVela’s plug-in management page is shown in Figure 8 below, including KubeVela’s own UI function and OCM multi-cluster function, which are all Plug-ins of KubeVela. On the basis of microkernel, users can customize and switch their own extension functions at will. At the same time, it also helps users quickly acquire more cloud native ecological capabilities, and helps enterprises better achieve resource management, DevOps, application governance, and other scenarios. At the implementation level, KubeVela’s plug-in system is completely based on the open source community model of co-construction and mutual assistance. All content submitted to the community plug-in repository **[6]** will be automatically discovered by the KubeVela kernel. We believe that there will be many options in the near future.
KubeVela also supports a number of component types by default, as shown in Figure 9, including application delivery based on container images. This pattern has a low threshold, requires no Kubernetes knowledge, and is suitable for most enterprise users. It also supports Kubernetes’ YAML applications, which are delivered to multiple clusters around OCM technology. In addition, users can obtain Helm Chart package, cloud resources of various cloud vendors and other component types by installing the officially maintained plug-ins, and obtain the corresponding complete application delivery and management capabilities. You can imagine what other types of applications KubeVela can deliver with this extension.
The concept of an environment is also added by default to KubeVela’s end-to-end UI architecture, supporting the different environments involved in the application development and delivery life cycle, such as development, test, production, etc. An application deployed in different environments is isolated, secure and reliable, and users do not need to configure it repeatedly, providing consistent user experience.
Currently, KubeVela 1.2 has been released as a preview version 1.2.0-beta.3**[7]**, and community users are welcome to download and experience it.
In the future, KubeVela will provide a more complete end-to-end user experience, enrich the capabilities in more vertical scenarios, and evolve towards self-delivery for developers, making App delivery in a hybrid environment as easy as it is today with the App Store.
A link to the
[1] CRD Operator:
Kubernetes. IO/docs/concep…
[2] CNCF Ecology:
landscape.cncf.io/
[3] Safety accident:
The about. Codecov. IO/security – up…
[4] Best Practices:
www.cncf.io/announcemen…
[5] White Paper:
www.cncf.io/announcemen…
[6] Community Plugin Repository:
Github.com/oam-dev/cat…
3: [7] 1.2.0 – beta.
Github.com/oam-dev/kub…
The original link
This article is the original content of Aliyun and shall not be reproduced without permission.