Hello, everyone. My name is Gong Chengming. I work in Tuling (Chengdu) Technology Co., LTD., mainly responsible for product system research and development and IT infrastructure construction of the company. This article will introduce our company in using KubeSphere platform to achieve the company’s business system container process.
Our company is a supplier of material resources for online template websites, providing template output and systematic solutions for customers. Help customers output standardized design products.
background
Migrating platform’s cloud native path
As early as 2020, the company’s IT team was relatively small, and development had to take part in operation and maintenance testing
In the early stage of development, development is basically driven by business. Based on resource factors, the system architecture first meets the function use and rapidly develops and launches products. The system architecture construction is also based on the evolution of Ali Cloud step by step from monomer to multi-module, and then to micro-service.
Private company business direction is the early printing goods ordering, personalize the output of the mobile terminal applications, supporting the supply of production order management system, at the same time involved in travel industry, provide custom circuit design SaaS system for travel agency, the output of the template posters system, as well as gallery travel required material resources.
Business pain points
After several years of development, business system services began to increase, and the basic technical architecture was difficult to cope with the rapid change of business. The R&D team also needed a reasonable development process to support the subsequent management.
We will mainly face difficulties for sorting out, roughly as follows:
- The development environment is inconsistent with the production environment
During project iterations, sometimes the configuration of the development and production environments is inconsistent, resulting in inconsistent production systems and business issues. 2. No unified release management system due to rough management of all aspects in the early stage, the lack of automatic construction system, after the version function, the development needs special manual compilation, packaging online release, the process is complex and difficult to manage. 3. Resource Coordination Although the business system has adopted the overall microservitization of SpringCloud, the allocation of various service resources cannot be coordinated. Printing services require several times more system resources than ordinary business systems to generate printed files, but they are not needed in real time. Before are specialized with a machine to do, but in fact this is not very flexible. So there is an urgent need for automatic capacity expansion and contraction.
Scheme selection
Based on the above pain points, combined with its own business system, ready to carry out container transformation.
When I first contacted Kubernetes, I learned about the management platform provided by the government. After investigation and trial, I found that it only manages the basic information of The Kubernetes container, and it is not simply put the business into the box, but involves the log platform and monitoring system of the business. The link and other basic operation and maintenance systems still need to introduce management by themselves. Finally, some integrated platform solutions, such as Rancher and KubeSphere, are suggested through the experience of our friends.
After comparison, we decided to adopt KubeSphere, mainly based on the following points:
- Kubernetes, a brand new knowledge system, has a high cost of time to master and achieve the implementation of production learning, so for our applied enterprises, we need products that can be easily used.
- Rancher focuses on operation and maintenance management, and the learning cost is relatively high. KubeSphere tends to focus on business applications, which is more in line with our company.
- Rancher needs to deploy plugins such as Jenkins itself; KubeSphere does a better job of integrating some components, such as DevOps out of the box. Release deployment is what we need right now.
- KubeSphere is a product launched by Qingyun Technology in China, which is more in line with the habits of Chinese people and completely open source.
Practice process
Existing hardware resources
The company’s entire business infrastructure is built on Ali Cloud, including ECS, database and OSS storage, etc.
The 6 ECS are distributed as follows:
- Ecs-1 to ECS-4: indicates service services.
- Ecs-5: Test machine.
- Ecs-6: Internal project management, including Bug management, Git, etc.
We will mainly implement the following steps:
- Build mirror warehouse on ECS-6, build Harbor warehouse. Provides a private image management tool for enterprise business container applications.
- Build a service system image Add the corresponding configuration file Dockerfile for each service to build an image when the platform pipeline is released.
- Prepare system environment system environment is mainly built by Kubernetes, here mainly consider storage and network selection.
-
Storage initially considered using Ceph, after building demo, found that if Kubernetes and Kubernetes built in the same cluster environment, there is a certain consumption of resources.
Based on the current service design (basically no stateful services are involved) and the current service volume, the lightweight NFS share disk is adopted.
-
The main network plug-ins of Kubernetes include Calico and Flannel. We chose Calico based on the experience of the community.
-
Install KubeSphere platform KubeSphere platform is based on Kubernetes according to the documentation provided on the official website.
We start with minimal scaffolding and then turn on some components as needed during use.
The KubeSphere platform has a better experience with plug-in installation, which can be easily implemented by adjusting the configuration file accordingly.
For example, the log platform is stored by Elasticsearch by default, but we have built our own Elasticsearch cluster, we just need to adjust the KS-Installer configuration.
Of course, there may be some problems, but basically the KubeSphere community can find solutions.
Practice the conversation
The CI/CD release process is the focus of this revamp.
The DevOps project is a pluggable component of KubeSphere that provides a Jenkins based CI/CD pipeline that supports automated workflows including binary-to-image (B2I) and source-to-image (S2I).
KubeSphere DevOps provides a CI/CD pipeline out of the box and a graphical way to lower the learning barrier, so we adapted the example from the official website directly, using configuration files to build and publish on pipeline Pipleline.
- Environment to distinguish the
Our environment corresponds to projects in KubeSphere, distinguished by specifying corresponding configuration files in the pipeline.
- Front-end Node environment specified
Since the node.js version provided by KubeSphere platform by default is different from the version we need, we modified the node.js environment of the platform through Jenkins plug-in based on our own experience, and the corresponding version can be specified in the subsequent pipeline.
Note: This approach is a bit troublesome, may be through the pipeline to specify the mirror should also be able to meet, but not in practice.
As for log collection, KubeSphere platform provides FluentBit Operator, which runs on DaemonSet on all nodes of the cluster. FluentBit is uniformly deployed and configured, and the query method can meet the existing business. Only Elasticsearch we docked our environment.
Practice effect
It took almost a month to complete the containerization of the basic business system.
The containerized development process is significantly better than before:
- We use KubeSphere’s Namespace for development, isolation of testing from production, and fine-grained permission management by assigning permissions to different enterprise Spaces to different roles.
- Version launch is based on Kubernetes copy and probe control, basically can be released at any time without affecting the business.
- The basic structure of the company to automatic process.
The future planning
At present, service grid is still in the exploratory stage, and service governance (such as monitoring indicators and microservice flow control) is still in the trial stage.
With the subsequent improvement of business complexity, this piece still hopes to land quickly. Try to achieve service governance in KubeSphere platform to achieve separation of business and technology.
Some expectations:
- Although the product experience has tried to reduce the user threshold, but the introduction of a lot of new concepts cloud native, simple guidance, ordinary users are still difficult to control.
- It would be better if our document could optimize the practical description and professional concept explanation of product function points.
- At the same time, we also hope that more people can participate in the maintenance of the community and experience the fun of open source!
This article is published by OpenWrite!