A background

As Kubernetes ecosystem is widely used in many enterprises, many operational and peacekeeping developers will manage and maintain multiple Kubernetes clusters at the same time, such as: Development environment, test environment, pre-release environment, production environment, etc., at the moment if you are still in the different KubeconFig file to rename the distinction, to switch clusters, then you can read this article, detailed harvest, enjoy the silky smooth multi-cluster switch management.

The second principle

Our daily management of K8S, through Kubectl, its essence is a client and Kube-Apiserver communication, because Kube-Apiserver needs to do authentication authentication, so Kubectl utility needs KubeconFig file to store authentication information, Normally KubeconFig stores the following information:

  1. Used to verifykube-apiserverCA root certificate of
  2. Used to identifykubectlThe administrator of theCertificate & Private key, or to identify ordinary userstoken

Kubeconfig is a configuration file in YAML format with the following main fields:

  • clustersThe type is array, and each element represents a K8S cluster
  • usersThe type is array, and each element represents a user with access rights
  • contextsThe type is array, and each element represents the one to be usedcluster & usercombination
  • current-contextThe context that is currently in use

Kubectl manages multiple clusters, essentially recording all the relevant parameters of the cluster and the relevant information of the user, and then combining them through the context, using the current-context parameter to indicate the current context.

3 Solutions

In order to facilitate the management of multiple environment of the cluster, is usually in the local environment through Kubernetes client tool Kubectl to manage multiple Kubernetes cluster. The first thing you want to do is manually merge multiple KubeconFig configuration files into one. How about using tools like Kubectx or Kubie to quickly switch context to manage multiple cluster environments? While it is possible to merge the KubeconFig configuration files manually, this method can be cumbersome when there are many cluster environments or when the cluster environment changes frequently.

3.1 Manual Merge

Currently I have the KS dev and Prod environment kubeconfig, so I can do the merge manually, the following files are merged manually

# apiversion
apiVersion: v1
Cluster field, which records multiple clusters as a list
clusters:
# ksprod cluster
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcxxx xxxxxxxxxxxxxxx
    server: https://lb.kubesphere.local:6443
  name: ksprod
# kstest cluster
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcxxx xxxxxxxxxxxxxxx
    server: https://172.16.60.2:6443
  name: kstest
The # context field is used to associate clusters with users
contexts:
# prodadmin@ksprod
- context:
    cluster: ksprod
    user: prodadmin
  name: prodadmin@ksprod
# testadmin@kstest
- context:
    cluster: kstest
    user: testadmin
  name: testadmin@kstest
The current cluster is that
current-context: prodadmin@ksprod
kind: Config
preferences: {}
# user field, list form records users
users:
# prod environment user
- name: prodadmin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJUGNvNy9qRzBOSkF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFRxxx xxxxxxxxxxxxxxx
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBMDhNMDFWOE5tTXliUUI2bGY4QlAzc1JCZWxhaWJOa2lCQStqUkY2Rxxx xxxxxxxxxxxxxxx
# test environment user
- name: testadmin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJR1Z3ZXNNTVRDV2t3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFRxxx xxxxxxxxxxxxxxx
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBeTVPT20zWEpERTlYTDUzWjh5UFhEQWMybmg0bkFzUDl2NW8xTUNZMxxx xxxxxxxxxxxxxxx
Copy the code

3.2 Official Mode

Officials currently provide two ways to solve this problem: configuring environment variables and specifying them through command line parameter displays.

3.2.1 Configuring Environment Variables

Configure the environment variable to specify kubeconFig files for multiple clusters

Configure Kubernetes multiple clusters, note the use of: separated
export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config:$HOME/.kube/mike-local-kubernetes.yaml:$HOME/.kube/dev-kubernetes.yaml:$HOME/. kube/test-kubernetes.yaml:$HOME/.kube/prod-kubernetes.yaml:$HOME/.kube/okteto-kube.yaml
Copy the code

3.2.2 Cli Parameters Specifying the Configuration file

Displays kubeconfig files specifying different clusters with command line arguments

# Switch to local cluster
kubectl get pod --kubeconfig=$HOME/.kube/mike-local-kubernetes.yaml

# Switch to the development cluster
kubectl get pod --kubeconfig=$HOME/.kube/dev-kubernetes.yaml

Switch to the test cluster
kubectl get pod --kubeconfig=$HOME/.kube/test-kubernetes.yaml

Switch to the production cluster
kubectl get pod --kubeconfig=$HOME/.kube/prod-kubernetes.yaml
Copy the code

Although the official method, can achieve multiple cluster management. However, it is obviously a hassle to constantly switch back and forth between kubeconFig profiles or constantly manually edit environment variables to add or subtract profiles from multiple clusters.

Now, this is the focus of this article, which is a more elegant solution.

3.3 Automatic merge tool

Since manual merging would be too cumbersome and complicated, is there any automatic merging tool? Don’t tell me, there really is and not this one. Let’s take a look at how to use these useful automatic merge tools.

3.3.1 kubeCM

3.3.1.1 Tool Overview

KubeCM is a use of Go language development KubeConfig management tool, very powerful function. It can not only achieve a number of KubeConfig file automatic merge, but also very convenient management of multiple Kubernetes cluster environment, such as: add, delete, rename different cluster environment.

3.3.1.2 installation

  • Binary installation
#According to your actual situation, download the corresponding platform binaries.

# Linux
$The curl - Lo kubecm. Tar. Gz https://github.com/sunny0826/kubecm/releases/download/v0.8.0/kubecm_0.8.0_Linux_x86_64.tar.gz

# macOS
$The curl - Lo kubecm. Tar. Gz https://github.com/sunny0826/kubecm/releases/download/v0.8.0/kubecm_0.8.0_Darwin_x86_64.tar.gz

# Windows
$The curl - Lo kubecm. Tar. Gz https://github.com/sunny0826/kubecm/releases/download/v0.8.0/kubecm_0.8.0_Windows_x86_64.tar.gz

#Unpack the

# Linux & macOS
$ tar -zxvf kubecm.tar.gz kubecm
$ sudo mv kubecm /usr/local/bin/

# Windows
#You can use any of the compression software to decompress and put$PATH 中
$ unzip kubecm.tar.gz
Copy the code
  • MAC one-click installation
$ brew install sunny0826/tap/kubecm
Copy the code

3.3.1.3 use kubeCM

For demonstration purposes, let’s create a kubeconFigdir directory and copy the cluster configuration files there.

$ ll kubeconfigdir- RW-r --r-- 1 xuel staff 2.1k 7 20 11:05 eks -- config-rw-r --r-- 1 xuel staff 5.3K 7 20 11:05 ks-prod-config-rw-r --r-- 1 Xuel Staff 5.3K 7 20 11:05 Ks-test-config-rw-r --r-- 1 xuel staff 7.1K 7 20 11:05 Smartops-config-prod-RW-r --r-- 1 xuel staff 5.3K 7 20 11:05 Smartops-config-prod-RW-r --r-- 1 Staff 5.3K 7 20 11:06 Smartops-config-testCopy the code

Note: As you may have noticed, my Kubeconfig configuration files end with.yaml to make it easier for the Kubie tool to switch between clusters. You only need to manage the Kubeconfig file according to the rules of your environment.

  • Automatic merge using KubeCM
#Merges all kubeconFig configuration files in the specified directory into one
$ kubecm merge -f mike-kubeconfig

#Merges all kubeconfig profiles in the specified directory into one and overwrites the default Kubeconfig profile

$ kubecm merge -f kubeconfigdir -c
Loading kubeconfig file: [kubeconfigdir/eks-config kubeconfigdir/ks-prod-config kubeconfigdir/ks-test-config kubeconfigdir/smartops-config-prod kubeconfigdir/smartops-config-test]
Context Add: eks-config
Context Add: ks-prod-config
Context Add: ks-test-config
Context Add: smartops-config-prod
Context Add: smartops-config-test
Copy the code

Note: the difference between using the -c parameter and not using the -c parameter is that the merged file name is.kube/config, while the merged file name is.kube/config.yaml.

  • Quickly add a cluster configuration using KubeCM
#Quickly add a cluster configuration file using KubeCM
$ kubecm add -f mike-local-kubernetes.yaml

#Use KubeCM to quickly add a cluster configuration file and specify the default namespace
$ kubecm add -f mike-local-kubernetes.yaml -n test
Copy the code
  • Use KubeCM to quickly remove a cluster configuration
#Delete a cluster configuration in command line mode
$ kubecm delete mike-local-kubernetes
Copy the code
  • Use KubeCM to quickly rename a cluster configuration
#Rename dev totest
$ kubecm rename -o dev -n test

#Rename the current context to dev
$ kubecm rename -n dev -c
Copy the code

The above are some general operations under KubeCM command line, in order to use the command line more efficiently. KubeCM also provides SHELL auto-completion, which can be easily set up by following the steps below.

  • Command completion

    • bash
    $ kubecm completion bash > ~/.kube/kubecm.bash.inc
$ printf "
# kubecm shell completion
source '$HOME/.kube/kubecm.bash.inc'
" >> $HOME/.bash_profile
$ source $HOME/.bash_profile
Copy the code
    • zsh
    # add to $HOME/.zshrc 
source <(kubecm completion zsh)
# or
$ kubecm completion zsh > "${fpath[1]}/_kubecm"
Copy the code

    KubeCM not only provides the command line mode, it also provides a more human interaction mode.

  • Interactively switch clusters with NS

Four reflection

At this point, we can easily switch between multiple clusters locally, and switch between multiple Ns for interactive management, and can easily add and remove other clusters. Of course, there are many excellent tools, I will not list them here.