Before the speech

Rancher as an open source enterprise Kubernetes cluster management platform. You can import existing clusters such as ACK, TKE, EKS, GKE, or use RKE, RKE2, K3s to customize deployment clusters.

As the industry’s leading multi-cluster management platform, Rancher can manage thousands of clusters and tens of thousands of nodes simultaneously. At the same time, there is no need to worry about the additional burden of operation and maintenance management of large-scale clusters and nodes. The social communication software LINE can manage 2000 nodes of 130 clusters with 5 people.

How to manage large clusters and nodes in Rancher is a topic that has been covered many times before. Today we’ll focus on managing large-scale projects, namespaces, and other resources in a single cluster.

How do I manage large-scale projects and namespaces in a single cluster

  • Namespaces: Namespaces are a Kubernetes concept that allows the creation of a virtual cluster within a cluster, which is useful for dividing clusters into separate “virtual clusters”, each with its own access controls and resource quotas.

  • Project: A project is a set of namespaces, a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, for example by setting up a team to have access to one project in the cluster but not to other projects in the same cluster.

Hierarchically, clusters contain projects, and projects contain namespaces. If a company is thought of as a cluster, then projects are departments or teams, and we can assign a department or team to manage multiple namespaces for different projects. So, we create a lot of projects and namespaces in production, so how do we plan them? It is recommended that projects be divided by department or team, and that access control be granted for each project

Namespaces are used to isolate environments with different purposes within each project, such as production and test environments

To verify the ability to manage large-scale projects and namespaces in a single Rancher cluster, we created 500 projects in a cluster, 10 namespaces in each project, and finally, created in each namespace: 1 Deployment, 2 Secret, 1 Service, and 2 ConfigMaps to simulate real-world usage.

Here is the test cluster size:

Thus we created a total of **500+ projects, 5000+ namespaces, 5000+ Deployment/ POD, 10000+ ConfigMap, 10000+ Secret, and 5000+ Service ** resources in a single cluster.

Next, let’s experiment with managing these resources through the Rancher UI:

First, log in to Rancher UI. Since there is only one cluster, no matter how many resources there are in the cluster, the load speed is the same as that of a normal sized cluster

When there are many items, we can quickly locate the items we need to query in the search box:

Enter a project and load it in about 1.5 seconds

Next, let’s create a Deployment that, from the results, loads as fast as a normal-sized cluster

Other resource management is not to show you one by one, test results and normal size cluster load speed is almost the same. From the above scenario analysis, a single cluster containing large-scale projects, namespaces, and other resources does not put too much performance pressure on Rancher. Due to the limited host resources in the test environment, only 5000 deployments were created for this verification. As long as resources are properly allocated to the project, it is believed that Rancher can easily manage several times more resources.

An unrecommended scenario

This scenario is not recommended, but it should be noted to avoid pitfalls when managing large projects and namespaces in a single cluster.

It is not recommended to stack all resources in one project, such as 5000+ namespaces in one project.

Why not? Simply put, when you enter a project from the Rancher Cluster Manager, all the resources contained in the project are loaded. The more resources a project contains, the slower the loading speed of resources in the project will be. If the amount of data is very, very large, the situation of timeout will sometimes occur. The Rancher management plane communicates with the API of the downstream cluster through the Cluster tunnel. If the number of resources is too large, the burden of the Websocket tunnel will be heavy.

While it is not recommended that you pile all your resources into the same project, it does not mean that this scenario cannot be managed in Rancher. If you have a production environment where you have to put all of your namespaces and other resources in the same project, and the amount of data is huge. It is recommended that you use Rancher V2.5’s new Cluster Explorer, which is optimized for loading resources within a single project, with almost no timeouts.

The following figure shows an example of loading 3000 namespaces, 3000 Deployment, 3000 Services, 6000 ConfigMap, 6000 Secret in the same Cluster Explorer project:

As a result, the Cluster Explorer, a new Dashboard for Rancher 2.5, can also manage thousands of resources in a single project.

Tuning recommendations for large cluster management

As a multi-cluster management platform, the increase of the workload of a single cluster or the number of clusters will pose certain performance challenges to the management plane. The default parameters of all components are the optimal product configuration of the common scale. However, for large-scale scenarios, some platform parameters and even system kernel parameters need to be tuned so that the core of the management plane can achieve optimal performance.

Users can take the corresponding optimization scheme according to their own scale of use. All optimization measures are not necessary and need to be tailored to their own scenes. Rancher’s default configuration parameters are already the best solution for most scenarios, and may need to be adjusted if you are managing large clusters. Here are some common optimization parameters for large clusters: In view of the large number of TCP connections on the management plane, the OS Kernel layer is optimized to maximize the performance of the hardware configuration. Common configurations and their functions are described as follows:

The following parameters are personal summary for reference only

Kernel Tuning

Kube-apiserverConcurrency and cache optimization for native resources and CRD apis to improve performance of API List-watch:

Kube-controller-manager, kube-Scheduler, kubelet, kube-proxy, kube-proxy, kube-proxy, kube-proxy, kube-proxy, kube-proxy, kube-proxy, kube-proxy, kube-Proxy, kube-Proxy, kube-Proxy, kube-Proxy

Remember after

From the above verification results, Rancher has the ability to manage large-scale projects and namespaces in a single cluster. As long as the cluster is built and deployed in a reasonable and correct way, Rancher will not bring too much pressure, nor will there be performance problems. Also tune the host and Kubernetes parameters for optimal performance based on the cluster size.

The authors introduce

Hailong Wang, community Technical Manager of SUSE Rancher, is responsible for the maintenance and operation of Rancher China technical community. I have 6 years of experience in cloud computing and experienced the technical transformation from OpenStack to Kubernetes. I have rich practical experience in operation and peacekeeping no matter the underlying operating system Linux, virtual KVM or Docker container technology.