Kubernetes GitOps tools
preface
In this article, I’ll review my favorite GitOps tools. In my opinion, the main advantage of Kubernetes is its declarative nature, and the main advantage of Kubernetes is its declarative nature and control loop, which continuously observes the real-time state of the cluster to ensure that it is connected to the storage in. NET ideal state matching. Etcd. The idea itself is very powerful, but it is limited to databases because it does not provide sufficient visibility. Etcd database, which does not provide sufficient visibility. This limits the liability and auditability of changes applied to the cluster. Another disadvantage is that we end up with two sources of truth, the ETCD and the codebase, which can lead to configuration drift, which is difficult to manage.
Developers have long used code libraries to easily store code in a secure and fully traceable manner. Workflow To effectively manage the central repository, different team members can work in parallel without too much friction, while ensuring that any changes are reachable, traceable, and easy to roll back.
Wouldn’t it be great if we could take all of these great advantages from the process created around Git repositories and extend them to the infrastructure? How about Kubernetes? The world of GitOps!
First, I’ll talk about what GitOps is and how to apply it to Kubernetes, then move on to declarative GitOps tools, which are used to implement GitOps in Kubernetes, and finally review some GitOps friendly tools, which are declarative and implemented in code.
What are GitOps?
The goal of GitOps is to create a single source of truth by extending this declarative feature from ETCD all the way to the Git repository where the code resides. By doing so, we also get all the benefits of a Git process, such as code review, history, fast rollback, traceability, and so on.
The core idea of GitOps is to have a Git repository that always contains a declarative description of the infrastructure currently required in the production environment, and an automated process to match the production environment to the described state in the repository. If you want to deploy a new application or update an existing one, you just update the repository — the automated process takes care of everything else. It’s like managing cruise control for your application in production. -www.gitops.tech/
GitOps is not unique to Kubernetes; in fact, it was created to extend the best practices of application code to the infrastructure as well, by storing it in the infrastructure, code in Git Repositories. This is popularized by tools such as Terraform. Note that “declarative infrastructure as code” plays a huge role in implementing GitOps, but it’s not just that. GitOps applies an entire ecosystem and tools around Git to the infrastructure. Simply putting the Terraform file in Git does not guarantee that the state of the infrastructure will always be the same as the state of the production environment. You’ll need to constantly run Terraform apply to see the live version, you’ll need to implement manual approval in your pipeline, and much more.
The idea of GitOps is to continuously observe the state of the cluster and compare it to a Git repository to make any immediate changes in the event of configuration drift. In addition, you get all the benefits of Git, such as code review for human approval.
To me, this idea is revolutionary and, if done right, will enable enterprises to focus more on functionality rather than scripting for automation. This concept can be extended to other areas of software development, for example, you can store your documents in code to track the history of changes and ensure that documents are up to date; Or use the following method to track architecture decision ADRs.
The GitOps Kubernetes
Kubernetes has adopted the idea of a control loop from the beginning, which means that Kubernetes is always watching the state of the cluster to ensure that it matches the desired state, for example, the number of replicates running matches the desired number of replicates. The idea behind GitOps is to extend it to applications, so you can define services as code, for example by defining helmsman diagrams, and use a tool that leverages K8s capabilities to monitor the state of your application and adjust the cluster accordingly. That is, if you update your code base, or your helmsman diagram, the production cluster will also be updated. This is really continuous deployment. The core principle is that application deployment and lifecycle management should be automated, auditable, and easy to understand.
The idea is to have a code base in each environment where you can define the ideal state for a given cluster. The Kubernetes operator then continues to monitor a particular branch (usually the master branch), and when a change is detected in Git, it is propagated to the cluster and the status is updated in etCD. In this way, ETCD serves only as a database, rather than as a single truth source for a cluster. In Git repositories containing the declarative Kubernetes infrastructure, you can have Helm chart definitions to define your application. In addition, you can chain warehouses, with one monitoring another, and so on. Kubernetes’ GitOps tool can monitor other repositories, such as the Helm Chart repository, so that your cluster environment repository does not need to have the Helm chart itself, but instead has a link to the Helm chart repository, which is also monitored for changes, so that when you publish a new chart, it is automatically deployed to the cluster. This way, you can have automatic end-to-end declarative CI/CD pipes.
Declarative GitOps tools
If we’re going to talk about GitOps in Kubernetes, we should start by talking about the tools that implement GitOps principles in Kubernetes, which are responsible for observing Git’s state and synchronizing it with the cluster.
ArgoCD
In my opinion, the best GitOps tool in Kubernetes is ArgoCD. You can read more about it here. ArgoCD is part of the Argo ecosystem, which includes some other great tools, some of which we’ll discuss later.
With ArgoCD, you can own each environment in a code base where you can define all the configurations for that environment. The Argo CD automatically deploys the required application state in the specified target environment.
ArgoCD architecture
The Argo CD is implemented as a Kubernetes controller that continuously monitors the running application and compares the current real-time state to the desired target state (as specified in Git repo). The Argo CD reports and visualizes differences and automatically or manually synchronizes the real-time state back to the desired target state.
The operation of the ArgoCD
ArgoCD has an excellent user interface that supports SSO, is secure, scalable, and very easy to use.
Flux
Flux is another very popular alternative to ArgoCD. The new version includes many improvements and features very similar to ArgoCD. Flux is an incubator project of CNCF.
The characteristics of the Flux
GitOps tools
In this section, we’ll review some of my favorite GitOps-friendly tools. In short, any tool based on custom resource definition (CRD) should be GitOPs-friendly.
The helmsman
Helmsman needs no introduction, it is Kubernetes’ most famous package manager. Yes, you should use the package manager in K8s just like you would use it in a programming language. Helm allows you to package your applications in diagrams. It abstracts complex applications into simple reusable components that are easy to define, install, and update.
It also provides a powerful templating engine. Helm is mature, with many preset charts, great support, and easy to use.
Helm is a perfect match for ArgoCD or Flux, as both can observe the Helm repository and deploy when a new chart is published. This works by pointing ArgoCD or Flux to a Helm repository and specifying a specific version or wildcard version. If you use wildcards, they will be automatically deployed whenever a new version is released. You can use major or minor versions. I usually like to package my applications in charts, build them as part of CI/CD, and then have ArgoCD monitor a specific repository. This separation of concerns allows developers to manage their applications in a separate repository, independent of their environment, and then let ArgoCD choose which charts are deployed to which environment. You can use several Helm repositories and perform promotions for different environments. For example, after a PR is merged, you can do a “Bronce” build and publish it as a Helm chart to the “Bronce” repository. The development environment’s ArgoCD repository points to the “Bronce” repository and deploys the new version as it becomes available. The ArgoCD repository in the staging environment points to the “silver” repository, while the production environment points to the gold repository. When you want to push something into staging or production, your CI/CD just needs to publish the chart to the next warehouse.
ArgoCD can override Helm values for any particular environment.
Customization is one of the newer, great alternatives to Helm, which does not use a template engine, but an override engine, where you have basic definitions and overlays on top of them.
Argo workflow and Argo events
In Kubernetes, you may also need to run batch jobs or complex workflows. This could be part of your data pipeline, asynchronous flow or even CI/CD. From there, you may need to run or even drive microservices that respond to certain events, such as files being uploaded or messages being sent to queues. For all of this, we have Argo workflows and Argo events.
Although they are independent projects, they are often deployed together.
Argo Workflows is a coordination engine similar to Apache Airflow, but originating in Kubernetes. It uses custom CRD to define complex workflows, uses steps or DAGs to use YAML and feels more natural in K8s. It has a nice user interface, retry mechanism, Cron based work, input and output handling, and more. You can use it to coordinate data pipelines, batch jobs, and more.
Sometimes, you might want to integrate your pipes with asynchronous services, such as streaming engines (such as Kafka), queues, Webhooks, or deep storage services. For example, you might want to react to events like files uploaded to S3. To do this, you will use Argo events.
Argo event
Together, these two tools provide a simple and powerful solution for all your plumbing needs, including CI/CD plumbing, which will enable you to run your CI/CD plumbing natively in Kubernetes.
As you might expect, Argo Workflows integrates perfectly with ArgoCD, because all workflow definitions are CRD and can be packaged in Helm diagrams.
For ML pipelines, you can use Kubeflow, which is built for this purpose.
For CI/CD pipelines, you can use Tecton.
Istio
Istio is the most famous service network in the market. NET, which is open source and very popular. I won’t go into detail about what service networking is, because it’s a huge topic. Microservices Then you will need a network of services to manage communication, observability, error handling, security, and all the other intersecting aspects that are part of the microservices architecture. Rather than contaminating the code of every microservice with repetitive logic, leverage the web of services to do it for you.
Istio architecture
In short, a service network is a specialized layer of infrastructure that you can add to your applications. It allows you to transparently add features like observability, traffic management, and security without having to add them to your own code.
Istio uses CRD for all of its functions, so virtual services, gateways, policies, and so on can all be defined as code, packaged as Helm diagrams, and used with ArgoCD or Flux, making Istio not only very powerful, but also GitOPs-friendly.
Linkerd or Consul are the best substitutes for Istio.
Argo promotion
We already mentioned that you can use Kubernetes to run your CI/CD pipes, use Argo workflow or similar tools. The next logical step is to continue with continuous deployment. This can be very challenging in a real world scenario because of the high stakes involved, which is why most companies just do continuous delivery, which means they have automation in place, but they still have manual approval and validation, and this manual step is due to the team not being able to fully trust their automation.
So how do you build that trust so that you can get rid of all scripts and fully automate everything from source code all the way to production? The answer is: observability. You need to focus your resources more on metrics and collect all the data you need to accurately represent the state of your application. Our goal is to use a set of metrics to build that trust. If you have all the data Prometheus then you can automate your deployment because you can automatically roll out your application based on those metrics.
In short, you need a more advanced deployment technique than what K8s offers, namely rolling updates. We need progressive delivery using canary deployment. Our goal is to progressively route traffic to the new version of the application, wait for metrics to be collected, analyze them, and match them to pre-defined rules. If everything works, we increase traffic; If there are any problems, we’ll go back to deployment.
To do this in Kubernetes, you can use the Argo scroll update which offers the Canary version and more.
Argo Rollouts is a Kubernetes controller and a CRD suite that provides Kubernetes with advanced deployment capabilities such as Blue green, Canary, Canary Analysis, experimental and incremental delivery capabilities.
While a service grid like Istio provides Canary publishing, Argo Rollouts makes the process easier and developer-centric as it is built specifically for this purpose. In addition, Argo Rollouts can be integrated with any service grid.
Argo Rollouts is GitOPs-friendly and perfectly integrates with Argo Workflows and ArgoCD ** ** Combining these three tools, you can create a very powerful declarative toolset that meets all of your deployment needs.
The FlaggerArgo Rollouts are very similar to Argo Rollouts and are highly integrated with Argo Rollouts. Flux so if you’re using Flux, consider Flagger.
There’s a Crossplane
Crossplane, my favorite K8s tool, brings a key missing piece to Kubernetes: managing third-party services as K8s resources. This means that you can configure AWS RDS or GCP Cloud SQL in the same way that you configure databases in K8s, using THE K8s resources defined in YAML.
With crossplanes, there is no need to use different tools and methods to separate infrastructure and code. You can use K8s resources to define everything. That way, you don’t have to learn new tools like Terraform and keep them separate.
Crossplane is an open source Kubernetes plug-in that enables platform teams to assemble infrastructure from multiple vendors and provide higher level self-service apis for application teams without writing any code.
Crossplane extends your Kubernetes cluster, providing you with CRD for any infrastructure or management cloud services. In addition, it allows you to fully implement continuous deployment because, in contrast to other tools such as Terraform, Crossplane uses existing K8s features such as control loops that continuously observe your cluster and detect any configuration drift automatically acting on it. For example, if you define a managed database instance and someone manually changes it, Crossplane will automatically detect the problem and set it to the previous value. This leaves infrastructure as code and the GitOps principle.
Crossplane works well with Argo CD, which looks at the source code to ensure that your code base is the only source of truth and that any changes in the code are propagated to clusters and external cloud services.
Without Crossplane, you can only implement GitOps in your K8s service, not in your cloud service, without using a separate process, now you can do this, which is very nice.
Kyverno corvino
Kubernetes offers great flexibility to enhance agile autonomous teams, but great power comes with great responsibility. There must be a set of best practices and rules to ensure that workloads that comply with company policies and security requirements are deployed and managed in a consistent and cohesive manner.
There are several tools to do this, but none of them are native to Kubernetes… Until now. Kyverno is a policy engine designed for Kubernetes. Policies are managed as a Kubernetes resource. No new language is required to write policies. The Kyverno policy can validate, mutate, and generate Kubernetes resources.
A Kyverno policy is a collection of rules. Each rule consists of a [match] (https://kyverno.io/docs/writing-policies/match-exclude/), an optional clauses [provides] (https://kyverno.io/docs/writing-policies/match-exclude/) clause, And a validate (https://kyverno.io/docs/writing-policies/validate/), [mutate] (https://kyverno.io/docs/writing-policies/mutate/), Or [generate] (https://kyverno.io/docs/writing-policies/generate) clause. A rule definition can contain only one validate,mutate, or generate child node.
You can apply any kind of policy regarding best practices, networking or security. For example, you could force all your services to have labels, or all containers to run as non-root. You can view some examples of strategies here. Policies can be applied to an entire cluster or to a specific namespace. You can also choose whether you want to audit policies only or enforce policies to prevent users from deploying resources.
Kubevela
One problem with Kubernetes is that developers need to have a good understanding and understanding of platform and cluster configurations. Many would argue that the level of abstraction in K8s is too low, creating a lot of friction for developers who just want to focus on writing and sending applications.
The Open Application Model (OAM) was created to overcome this problem. The idea is to create a higher level of abstraction around the application that is independent of the underlying runtime. You can read the specification here.
The Open Application Model [OAM] focuses on applications, not containers or coordinators, and brings modular, extensible, and portable designs for modeling application deployments, with higher-level but consistent apis.
KubevelaKubeVela is an implementation of the OAM model. KubeVela is runtime agnostic, has native extensibility, but most importantly, is application-centric. In Kubevela, applications are first class citizens implemented as Kubernetes resources. There is a distinction between cluster operators (platform teams) and developers (application teams). Cluster operators define applications by defining components (deployable/providable entities that make up your application, such as helmsman diagrams) and attributes **.** Developers define applications by assembling components and attributes.
Platform team. Model and manage platform capabilities as components or attributes, as well as target environment specifications. Application team. Select an environment, assemble the application with components and characteristics as needed, and deploy it to the target environment.
KubeVela is a sandbox project of the Cloud Native Computing Foundation, and although it is still in its infancy, it could change the way we use Kubernetes in the near future, enabling developers to focus on applications rather than Kubernetes experts. However, I do have some concerns about the real-world applicability of OAM, as some services, such as system applications, ML, or big data processes, rely heavily on low-level details that may be difficult to incorporate into the OAM model.
SchemaHero Heroes
Another common process in software development is the evolution of management schemas when using relational databases.
SchemaHero is an open source database schema migration tool that transforms schema definitions into migration scripts that can be applied in any environment. It uses the declarative nature of Kubernetes to manage database schema migration. You just specify the required states, and SchemaHero manages the rest.
Bitnami sealed the secret
We’ve covered many GitOps tools, such as ArgoCD. Our goal is to put everything in Git and take advantage of Kubernetes’ declarative nature to keep the environment in sync. We’ve just seen how we can (and should) keep our source of truth in Git and let an automated process handle configuration changes.
One thing that’s usually hard to keep in Git is secrets like DB passwords or API keys, because ** you shouldn’t store secrets in a code repository. ** A common solution is to use external vaults such as AWS Secret Manager or HashiCorp vault to store secrets, but this creates a lot of friction because you need to have a separate process for handling secrets. Ideally, we’d like to have a way to safely store secrets in Git just like any other resource.
Bitnami Sealed Secrets was created to overcome this problem, allowing you to store your sensitive data in Git by using powerful encryption techniques. BitnamiSealed Secrets is native integrated in Kubernetes, only allowing The Kubernetes controller running in Kubernetes to decrypt Secrets, no one else. The controller will decrypt the data and create a local K8s secret for secure storage. This allows us to store everything as code in our REPO, allowing us to safely perform continuous deployment without any external dependencies.
Sealed Secrets consists of two parts.
- A cluster side controller
- A client tool.
kubeseal
The Kubeseal tool uses asymmetric encryption techniques to encrypt secrets that only the controller can decrypt. These encrypted secrets are encoded in a SealedSecret K8s resource that you can store in Git.
Capsule Capsule
Many companies use multi-tenancy to manage different customers. This is common in software development, but difficult to implement in Kubernetes. Namespaces are a good way to create logical partitions of a cluster as isolated slices, but this is not enough. To safely isolate customers, we need to enforce network policies, quotas, and so on. You can create network policies and rules for each namespace, but this is a cumbersome process that is difficult to scale. In addition, tenants will not be able to use more than one namespace, which is a major limitation.
To overcome some of these problems, we created a new K8 system. The idea is to provide a parent namespace for each tenant, with common network policies and quotas for tenants that allow the creation of child namespaces. This is a big improvement, but it has no local support for tenants in terms of security and governance. Also, it’s not in production status yet, but a 1.0 release is expected in the next few months.
A common solution to this problem today is to create a cluster for each customer, which is secure and provides everything tenants need, but it is difficult to manage and very expensive.
Capsule is a tool that provides native Kubernetes** support for multiple tenants within a single cluster. ** With Capsule, you can provide a cluster for all your tenants. Capsule will provide tenants with an “almost” native experience (with some minor limitations) and they will be able to create multiple namespaces and use clusters because it’s completely available to them, hiding the fact that clusters are actually shared.
Capsule structure
In a cluster, the capsule controller aggregates multiple namespaces into a lightweight Kubernetes abstraction called a tenant, which is a grouping of Kubernetes namespaces. Within each tenant, users are free to create their namespaces and share all allocated resources, while the policy engine maintains isolation between different tenants.
Network and security policies, resource quotas, limits, _RBAC_, and other policies defined at the tenant level are automatically inherited by all namespaces in the tenant, similar to hierarchical namespaces. Users are then free to operate their tenants autonomously without the intervention of a cluster administrator.
Capsule is ready for GitOps because it’s declarative and all of its configuration can be stored in Git.
conclusion
In this article, we review my favorite GitOps-friendly Kubernetes tools. I focused on open source projects that could be incorporated into any Kubernetes distribution.
My goal is to show you that, thanks to GitOps best practices, you can do everything in Kubernetes with its declarative nature.