VPGAME is a comprehensive esports service platform integrating event operation, media information, big data analysis, player community and game periphery. Headquartered in Hangzhou, China, it has set up esports big Data RESEARCH and development center in Shanghai and AI research and development center in Seattle. This article describes how VPGAME migrated its servers to Kubernetes.

background

As container technology matures, the company plans to migrate services to a container environment, scheduling, scheduling and managing containers through Kubernetes. It also takes this opportunity to standardize services, optimize the entire CI/CD process, and improve the efficiency of service deployment.

CI/CD tool selection

For CI/CD tools, we chose Gitlab-CI. Gitlab-ci is a continuous integration system that works with GitLab to install dependencies, compile, unit test, Lint, image build, and release code after submission.

Gitlab-ci is perfectly integrated with GitLab and only needs to install and configure Gitlab-Runner when it is used. After registering with GitLab, Gitlab-runner can provide an environment for CI/CD operation, and is responsible for pulling codes from GitLab and executing corresponding commands for CI/CD work according to gitlab-ci.yml configured in the code warehouse.

Compared with Jenkins, the configuration of Gitlab-CI is simple. It only needs to configure the gitlab-ci.yml file in the project to complete the compilation of CI/CD process, and there is no need to configure the Webhook callback address as in Jenkins. There is no need for Jenkins to create a new build configuration for the project. And I personally think GitLab’s CI/CD process display is more beautiful than Jenkins. Of course, Jenkins can configure many functions that gitlab-CI does not exist by relying on its rich plug-ins. According to our current requirements, Gitlab-CI is simple and easy to use, and its functions also meet our requirements.

Service runtime environment

Container environment advantages

Traditional way of service deployment is in the operating system installed the corresponding application dependent, and then in the installation of the application service, this way of deployment of the disadvantage is that the service process, configuration, dependent libraries, and life cycle and the host operating system, tightly coupled, enlarge shrinks to upgrade the service capacity and migration operation is not very convenient.

The container deployment mode is centered on the image. When the code is compiled and constructed, the application program and the dependencies required by the application program are packaged into an image. In the deployment phase, the container instance is created through the image to complete the deployment and operation of the service. In this way, application-centered management is realized, and container isolation achieves resource isolation. Because containers do not need to depend on the operating system environment of the host, the consistency of development, testing and production environments can be well guaranteed. In addition, because built images are immutable and versioned with tags, they provide reliable, frequent container image builds and deployments, as well as easy and fast rollback operations.

Kubernetes platform features

Kubernetes (K8S for short), as a container scheduling, scheduling, and management platform, can schedule and run application containers on physical or virtual machine clusters, providing a container-centric infrastructure. Kubernetes allows you to orchestrate and manage containers by:

  • Deploy services quickly and predictably
  • Ability to expand services on the fly
  • Rolling upgrade, complete the release of new functions
  • Optimize hardware resources and reduce costs
Ali Cloud container service advantages

We choose ali cloud container services in the service migration, it is based on the original Kubernetes for adaptation and enhancement, simplify the cluster construction and expansion work, integration of Ali cloud virtualization, storage, network and security capabilities, to create the cloud’s best Kubernetes container application running environment. In terms of convenience, you can create and upgrade The Kubernetes cluster as well as expand and shrink the node capacity through the Web interface. In terms of functions, it integrates with ari Cloud resources in terms of network, storage, load balancing, and monitoring to minimize the impact of migration.

In addition, when choosing cluster creation, we chose the managed version of Kubernetes, which simply creates the Worker node, while the Master node is created and hosted by the container service. In this way, we still have autonomy and flexibility in the planning and resource isolation of Worker nodes and do not need operation and maintenance management of Kubernetes cluster Master nodes, so we can focus more energy on application services.

GitLab Runner deployment

GitLab CI workflow





Basic concepts of GitLab CI

Before introducing GitLab CI, I would like to briefly introduce some basic concepts in GitLab CI as follows:

  • Pipeline: Pipeline in Gitlab CI. A Pipeline is generated each time a code submission triggers the Gitlab CI.
  • Stages: Each Pipeline consists of multiple stages, and each Stage is in sequence.
  • Job: The smallest unit of work in a GitLab CI that is responsible for doing one thing, such as compiling, testing, building an image, etc. Each Job needs to specify stages, so the Job execution sequence can be realized by specifying different stages.
  • GitLab Runner: the environment where a specific Job is executed. Each Runner can execute only one Job at a time.
  • Executor: Each Runner needs to specify an Executor when registering with GitLab to determine the type of Executor to complete the Job.
GitLab CI workflow

When code is pushed to GitLab, a Pipeline is triggered. Operations such as compiling, testing, and image building are performed, each of which is a Job. In the CD phase, the results built in the CI phase are deployed to the test environment or production environment, depending on the situation.

GitLab Runner is introduced

Gitlab Runner classification

There are three types of runners in GitLab, which are:

  • Shared: used by all projects
  • Group: used by projects under group
  • Specific: Specifies the use of the project
We can register different types of runners with GitLab according to our needs, and the way of registration is the same.

Gitlab Runner working process

Runner will first initiate a registration request to GitLab, which contains token, tag and other information. After successful registration, GitLab will return a token to Runner, and Runner will carry this request in subsequent requests.

After successful registration, Runner will continuously request Job to GitLab at an interval of 3s. GitLab returns 204 No Content if No Job was requested. If the request reaches the Job, GitLab will send the Job information back. After receiving the Job, Runner will send a confirmation request to GitLab and update the status of the task. After that, Runner starts Job execution and periodically sends intermediate data to GitLab in the form of Patch request.





The Executor GitLab Runner

The Runner actually performs the Job by calling Executor. Runner provides SSH, Shell, Docker, Docker-SSH, VirtualBox, Kubernetes and other types of executors to meet different scenarios and requirements during registration.

Among them, Shell and Docker executors are commonly used. Shell types mainly use the environment of the host of Runner to execute the Job. A Docker-type Executor pulls an image at the start of each Job to produce a container that completes the Job. When the Job completes, the container is destroyed. Because Docker is highly isolated, lightweight, and recyclable, we use docker-type executors to execute jobs. We only need to prepare Docker images of the environment required for jobs in advance and define images for each Job to use the corresponding environment, which is convenient to operate.

GitLab Runner installation and configuration

Docker installation

Because we need to use docker-type Executor, we need to install Docker on Runnner server first, the details are as follows (CentOS environment) :

Install the required package, yum-util provides yum-config-manager functionality, the other two are DeviceMapper driver dependencies:

yum install -y yum-utils device-mapper-persistent-data lvm2

Configure the yum source:

yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install the Docker:

yum install docker-ce -y

Start and join boot boot:

systemctl start docker systemctl enable docker

Gitlab Runner installation and startup

Execute the following commands to install and start GitLab Runner:

Packages.gitlab.com/install/rep ` curl – L… | sudo bash sudo yum install gitlab-runner -y gitlab-runner start`

GitLab Runner registration with configuration updates

After starting GitLab Runner, you also need to register with GitLab. Before registration, you need to query tokens from GitLab. Different types of runners use different paths to obtain tokens. Shared Runner requires an account with admin permission to obtain the corresponding token.





For the other two types, setting – >CI/CD – >Runner on the corresponding page (group or project home page) can obtain the token.

Runner registration can be divided into interactive and non-interactive methods. In interactive registration mode, after entering the gitlab-Runner register command, the information required for registration is entered as prompted, including gitlab URL, token and Runner name, etc. Personally, I recommend non-interactive commands. You can prepare commands in advance to complete one-click registration. In addition, non-interactive registration provides more registration options to meet diversified requirements.

To complete a Runner registration, follow the following example:

  1. gitlab-runner register –non-interactive \
  2. –url “http://git.xxxx.cn” \
  3. –registration-token “xxxxxxxxxxx” \
  4. –executor “docker” \
  5. –docker-image alpine:latest \
  6. –description “base-runner-docker” \
  7. –tag-list “base-runner” \
  8. –run-untagged=”true” \
  9. –docker-privileged=”true” \
  10. –docker-pull-policy “if-not-present” \
  11. –docker-volumes /etc/docker/daemon.json:/etc/docker/daemon.json \
  12. –docker-volumes /etc/gitlab-runner/key/docker-config.json:/root/.docker/config.json \
  13. –docker-volumes /etc/gitlab-runner/find_diff_files:/usr/bin/find_diff_files \
  14. –docker-volumes /etc/gitlab-runner/key/id_rsa:/root/.ssh/id_rsa \
  15. –docker-volumes /etc/gitlab-runner/key/test-kube-config:/root/.kube/config
–docker-pull-policy Specifies the Dokcer image download policy for Job execution by Executor. Docker-volumes specifies the file mount mapping between the container and the host. The files mounted above are mainly used for some keys used by runners to execute jobs, including keys to access GitLab, Docker Harbor, and Kubernetes clusters. If there are other files to share with the container, run –docker-volumes to add volumes.

The /etc/docker-daemon. json file is mainly set to allow HTTP access to docker horbor:

{ “insecure-registries” : [“http://docker.vpgame.cn”] }

After registration, restart Runner:

gitlab-runner restart

After the deployment, information about different runners can be viewed in the Web management interface of GitLab.





In addition, if a service needs to register multiple runners, you can change the concurrent value in /etc/gitlab-runner/config.toml to increase the number of concurrent runners. After the change, you also need to restart Runner.

Docker basic image making

In order to meet the diversified requirements of different services on the operating environment, we need to prepare different basic images for services of different languages in advance to be used in the mirror construction phase. In addition, the tool image required by CI/CD also needs to be made, as the Docker image needed by the container is generated when the Runner performs the Job.

All images are managed by GitLab in the form of writing dockerfiles, and we write.gitlab-ci.yml files, Every time a Dockerfile is added or modified, Pipeline will be triggered to build the image and upload it to Harbor. This management mode has the following advantages:

  • An image is automatically created based on certain rules, which allows you to quickly create and update an image
  • According to the rules, you can find the corresponding Dockerfile of the image, and make clear the specific composition of the image
  • Team members are free to build their own images by submitting the Merge Request

Image classification





  • Runtime base image: Provides the necessary tools and corresponding packages for each language runtime.
  • CI mirroring: Based on the run-time base mirroring, add unit testing, Lint, static analysis, etc., for the test part of the CI/CD process.
  • Package live images: used in the Build and deploy parts of the CI/CD process.

Dockerfile directory structure

Each folder has a Dockerfile to describe the basic situation of the image, including Java, PHP, Node and Go runtime base image and CI image, as well as tools such as Docker-Kubectl image Dockerfile.





Take PHP mirroring as an example: 

PHP / ├ ─ ─ 1.0 │ ├ ─ ─ Dockerfile │ ├ ─ ─ ci 1.0 │ │ └ ─ ─ Dockerfile │ ├ ─ ─. PHP ini │ ├ ─ ─read│ ├─ www.conf ├─ ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─ nginx ├─Copy the code
There is a folder called 1.0 that contains a Dockerfile used to build the PHP FPM runtime base for mirroring. We added our own custom files and specified the working directory and container initial commands in PHP: 7.1.16-FPM-Alpine3.4.

  1. The FROM PHP: 7.1.16 – FPM – alpine3.4


  3. RUN sed -i ‘s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g’ /etc/apk/repositories\
  4. && apk upgrade –update && apk add –no-cache –virtual build-dependencies $PHPIZE_DEPS \
  5. tzdata postgresql-dev libxml2-dev libmcrypt libmcrypt-dev libmemcached-dev cyrus-sasl-dev autoconf \
  6. && apk add –no-cache freetype libpng libjpeg-turbo freetype-dev libpng-dev libjpeg-turbo-dev libmemcached-dev \
  7. && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
  8. && echo “Asia/Shanghai” > /etc/timezone \
  9. && docker-php-ext-configure gd \
  10. –with-gd \
  11. –with-freetype-dir=/usr/include/ \
  12. –with-png-dir=/usr/include/ \
  13. –with-jpeg-dir=/usr/include/ \
  14. && docker-php-ext-install gd pdo pdo_mysql bcmath opcache \
  15. && pecl install memcached apcu redis \
  16. && docker-php-ext-enable memcached apcu redis \
  17. && apk del build-dependencies \
  18. && apk del tzdata \
  19. && rm -rf /var/cache/apk/* \
  20. && rm -rf /tmp/* \
  21. && rm -rf /working/* \
  22. && rm -rf /usr/local/etc/php-fpm.d/*


  24. COPY start_service.sh /usr/local/bin/start_service.sh
  25. COPY read-zk-config /usr/local/bin/read-zk-config
  26. COPY php.ini /usr/local/etc/php/php.ini
  27. COPY www.conf /usr/local/etc/php-fpm.d/www.conf
  28. WORKDIR /work
  29. CMD [“start_service.sh”]
There is also a Dockerfile in 1.0/ CI-1.0 that builds CI images that PHP uses for unit testing and Lint operations. You can see how it builds with additional tools based on the base runtime image above.

  1. The FROM docker. Vpgame. Cn/infra/PHP 1.0


  3. ENV PATH=”/root/.composer/vendor/bin:${PATH}”
  4. ENV COMPOSER_ALLOW_SUPERUSER=1


  6. RUN mkdir -p /etc/ssh && echo “StrictHostKeyChecking no” >> /etc/ssh/ssh_config
  7. RUN apk –update add –no-cache make libc-dev autoconf gcc openssh-client git bash &&\
  8. echo “apc.enable_cli=1” >> /usr/local/etc/php/conf.d/docker-php-ext-apcu.ini
  9. RUN pecl install xdebug && docker-php-ext-enable xdebug &&\
  10. echo -e “\nzend_extension=xdebug.so” >> /usr/local/etc/php/php.ini
  11. Wget RUN https://vp-infra.oss-cn-beijing.aliyuncs.com/gitlab-ci/software/download/1.6.5/composer.phar – O/bin/composer && \
  12. chmod +x /bin/composer && \
  13. composer config -g -q repo.packagist composer https://packagist.laravel-china.org
  14. RUN composer global require -q phpunit/phpunit:~5.0 squizlabs/php_codesniffer:~3.0


  16. WORKDIR /


  18. CMD [“/bin/bash”]
There is also a Dockerfile in the Nginx directory to customize the Nginx image for our PHP project.

When adding a new Dockerfile or changing the Dockerfile for the first time in GitLab, Pipeline will be triggered to automatically build the image and upload it to our private Docker Harbor.





Basic principles of image automatic construction

Git diff > git diff > git diff > git diff > git diff And upload it to Docker Harbor.

Finddifffiles in the above command finds the files that are different before and after the merge based on the git diff command.

Accelerate the tips

  • Alpine Linux Package Management (APK) mirrors.aliyun.com
  • Some overseas software will be slow to download, so you can download it first and upload it to Ali Cloud OSS for download. Dockerfile uses Ali Cloud OSS as the download source to reduce the time to build the image.

Ci /CD flow based on.gitlab-ci.yml

After the GitLab Runner and Docker base images are made, we can go through the CI/CD process to complete the unit testing, lint, compilation, image packaging, and deployment of code updates. To perform CI/CD operations through GitLab CI, you only need to edit and maintain a.gitlab-ci.yml file in the code repository. Whenever the code is updated, GitLab CI will read the contents of.gitlab-ci.yml. Generate a Pipeline for CI/CD operations. . Gitlab-ci. yml has a relatively simple syntax, and describes jobs based on YAML syntax. We divide the tasks to be completed in the CI/CD process into jobs in the file. As long as each Job is clearly defined, an appropriate, efficient and universal CI/CD process can be formed.

Define the stages

“Stages” is an important concept. It is defined globally in.gitlab-ci.yml. When defining a Job, a value is specified to indicate the stage of the Job. The order of elements in stages defines the order of Job execution. All jobs on the same stage are executed in parallel. Only when all jobs on the current stage are successfully completed, jobs on subsequent stages are executed.

For example, define stages as follows:

`stages:

  • build
  • test
  • deploy`
  1. First, all jobs in a build are executed in parallel.
  2. If all jobs in build are successfully executed, all jobs in test are executed concurrently.
  3. If all jobs in test are executed successfully, all jobs in deploy are executed in parallel.
  4. If all jobs in deploy are executed successfully, the current Pipeline is marked with passed.
  5. When a stage Job fails to execute, the Pipeline marks it as failed, and the jobs of subsequent stages are not executed.

The description of the Job

Job is the most important part of.gitlab-ci.yml file. All tasks performed in CI/CD processes can be realized by defining jobs. Specifically, we can describe each Job by keyword. Because there are so many keywords in Job and their usage is rich, here is a Job in our own actual combat to illustrate.

  1. unittest:
  2. stage: test
  3. Image: docker. Vpgame. Cn/infra/PHP – 1.0 – the ci – 1.1
  4. services:
  5. – name: docker. Vpgame. Cn/infra/mysql – 5.6 – multi
  6. alias: mysql
  7. – name: redis: 4.0
  8. alias: redis_default
  9. script:
  10. – mv .env.tp .env
  11. – composer install –no-dev
  12. – phpunit -v –coverage-text –colors=never –coverage-html=coverage –stderr
  13. artifacts:
  14. when: on_success
  15. paths:
  16. – vendor/
  17. – coverage/
  18. expire_in: 1 hour
  19. coverage: ‘/^\s*Lines:\s*\d+.\d+\%/’
  20. only:
  21. – branches
  22. – tags
  23. tags:
  24. – base-runner
The above Job performs the function of unit testing and defines the Job name in the start line. The following describes the specific meanings of each keyword of Job.

Stage: defines the stage of a Job. The value is defined in the global stages.

Image specifies the image that Runner needs to run. This image is the basic image we made before. The Docker that runs through this image is the environment in which the Job runs.

Services, the connection dependency required by the Docker run by Runner. MySQL and Redis are respectively defined here, and the Docker generated by these two images will be connected when the Job runs.

Script The specific command that the Job runs is described by Shell. The script in this Job does the compilation and unit testing of the code.

Artifacts is about wrapping results completed in the Job and saving them. You can use when to specify when to save the artifacts, path defines the file path to save, and expire_in specifies the validity period for saving the results. This corresponds to the Dependencies parameter. If other jobs require artifacts of this Job, follow the following definition on the Job.

`dependencies:

  • unittest`
The only keyword specifies the time when the Job is triggered. In this example, the Job is triggered only when branches are merged or a tag is added.

In contrast to only, there is the except keyword to exclude certain situations that trigger a Job. In addition, only supports regular expressions, such as:

  1. job:
  2. only:
  3. – /^issue-.*$/
  4. except:
  5. – branches
In this example, only the tag starting with issue- triggers the Job. Without the except argument, branches or tags starting with issue- trigger jobs.

Tags the tags keyword is used to specify the type of Runner to run. In our practical application, runners used in the deployment test environment and production environment are different, and they are identified and distinguished by different tags.





Therefore, in the Job definition, Runner is specified by tags to specify the required Runner.

We can see that the definition of Job is very clear and flexible. The usage of Job is far more than these functions. Please refer to the official documentation of GitLab CI/CD for more details.

CI/CD process choreography

< span style = “box-sizing: border-box! Important; word-wrap: break-word! Important;” < span style = “max-width: 100%; box-sizing: border-box! Important; word-break: inherit! Important;”

When PHP project Kubernetes went live, we specified that merging the Master branch would do four jobs: Lint, Unitest, build-test, and deploy-test.





After the test environment is verified, we will launch the formal environment by tagging. Pipelines in charge contain unittest, build-Pro, and deploy-Pro jobs.

The build stage:

  1. \# Build stage


  3. .build-op:
  4. stage: build
  5. dependencies:
  6. – unittest
  7. Image: docker. Vpgame. Cn/infra/docker kubectl – 1.0
  8. services:
  9. – name: docker:dind
  10. entrypoint: [“dockerd-entrypoint.sh”]
  11. script:
  12. – echo “Image name:” ${DOCKER_IMAGE_NAME}
  13. – docker build -t ${DOCKER_IMAGE_NAME} .
  14. – docker push ${DOCKER_IMAGE_NAME}
  15. tags:
  16. – base-runner


  18. build-test:
  19. extends: .build-op
  20. variables:
  21. DOCKER_IMAGE_NAME: ${DOCKER_REGISTRY_PREFIX}/${CI_PROJECT_PATH}:${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}
  22. only:
  23. – /^testing/
  24. – master


  26. build-prod:
  27. extends: .build-op
  28. variables:
  29. DOCKER_IMAGE_NAME: ${DOCKER_REGISTRY_PREFIX}/${CI_PROJECT_PATH}:${CI_COMMIT_TAG}
  30. only:
  31. – tags
In the build stage, the basic operation of image packaging in the test environment and production environment is the same, which is based on the Dockerfile to upload the image build and image warehouse. The extend parameter is used to reduce repetitive Job descriptions, making them more concise and clear.

We define a. Build-op Job, and then both build-test and build-prod inherit from extend. We can add or override the configuration in. Build-op by defining keywords. For example, build-prod redefines variables, DOCKER_IMAGE_NAME, and trigger conditions (only) to tag.

Another thing to note here is that when defining the DOCKER_IMAGE_NAME, we refer to some variables of the GitLab CI itself, such as CI_COMMIT_TAG, which represents the tag name of the project’s commit. We may refer to some GitLab CI Variables when defining Job Variables. Please refer to the Chinese documentation of GitLab CI/CD Variables for explanations of these Variables.

The deploy stage:

  1. \# Deploy stage
  2. .deploy-op:
  3. stage: deploy
  4. Image: docker. Vpgame. Cn/infra/docker kubectl – 1.0
  5. script:
  6. – echo “Image name:” ${DOCKER_IMAGE_NAME}
  7. – echo ${APP_NAME}
  8. – sed -i “s~__NAMESPACE__~${NAMESPACE}~g” deployment.yml service.yml
  9. – sed -i “s~__APP_NAME__~${APP_NAME}~g” deployment.yml service.yml
  10. – sed -i “s~__PROJECT_NAME__~${CI_PROJECT_NAME}~g” deployment.yml
  11. – sed -i “s~__PROJECT_NAMESPACE__~${CI_PROJECT_NAMESPACE}~g” deployment.yml
  12. – sed -i “s~__GROUP_NAME__~${GROUP_NAME}~g” deployment.yml
  13. – sed -i “s~__VERSION__~${VERSION}~g” deployment.yml
  14. – sed -i “s~__REPLICAS__~${REPLICAS}~g” deployment.yml
  15. – kubectl apply -f deployment.yml
  16. – kubectl apply -f service.yml
  17. – kubectl rollout status -f deployment.yml
  18. – kubectl get all,ing -l app=${APP_NAME} -n $NAMESPACE


  20. \# Deploy test environment
  21. deploy-test:
  22. variables:
  23. REPLICAS: 2
  24. VERSION: ${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}
  25. extends: .deploy-op
  26. environment:
  27. name: test
  28. url: http://example.com
  29. only:
  30. – /^testing/
  31. – master
  32. tags:
  33. – base-runner


  35. \# Deploy prod environment
  36. deploy-prod:
  37. variables:
  38. REPLICAS: 3
  39. VERSION: ${CI_COMMIT_TAG}
  40. extends: .deploy-op
  41. environment:
  42. name: prod
  43. url: http://example.com
  44. only:
  45. – tags
  46. tags:
  47. – pro-deploy
Similar to the build phase, a. Deploy-op Job is defined, and deploy-test and deploy-prod are inherited by extend.

Deploy-op mainly completes the substitution of some variables in the Kubernetes Deployment and Service template files and the Deployment of Kubernetes services from the generated Deployment and Service files.

The deploy-test and deploy-prod jobs define different variables and trigger conditions (only). In addition, deploy-Prod uses the tags keyword to use different runners to point the deployed target cluster to Kubernetes in the production environment.

There is one more keyword that needs to be noted here: after environment is defined, we can view some information for each deployment in GitLab. In addition to seeing some information about each deployment, we can easily redeploy and roll back.





As you can see, by configuring the keywords of the Job, we can flexibly arrange the CI/CD flow that we need, which is very suitable for diverse scenarios.

Deployment and Service configuration

After the Docker image packaging task is completed in CI/CD process, the image corresponding to the service needs to be deployed to Kubernetes cluster. Kubernetes provides a variety of resource objects that can be scheduled. First, let’s take a quick look at some of the basic resources in Kubernetes.

Overview of Kubernetes base resource objects

Pod

Pod, as the running entity of stateless applications, is one of the most commonly used resource objects, and the smallest basic unit of resource scheduling in Kubernetes. It contains one or more closely related containers. These containers share storage, networks, and namespaces, as well as specifications for how to operate.

In Kubernetes, pods are non-persistent and can be destroyed and rebuilt due to node failure, network failure, etc. So in Kubernetes, we don’t create a single Pod directly, but provide services through multiple pods.

ReplicaSet

ReplicaSet is a replica controller in Kubernetes. It controls the managed PODS and keeps the number of Pod replicas to a preset number. ReplicaSets can be used independently, but in most scenarios is used by Deployments as a mechanism to coordinate Pod creation, deletion, and update.

Deployment

Deployment provides a declarative definition method for Pod and ReplicaSet. By describing the target state in Deployment, the Deployment Controller changes the actual status of the Pod and ReplicaSet to the specified target state. Typical Deployment scenarios include:

  • Define Deployment to create Pod and ReplicaSet
  • Rolling updates and rolling back applications
  • Capacity expansion and reduction
  • Suspend and continue Deployment
Service

In Kubernetes, pods can be created or destroyed at any time. Each Pod has its own IP and these IP cannot persist, so services are needed to provide Service discovery and load balancing capabilities. A Service is a policy abstraction that defines a set of PODS and provides an entry point for clients to access the Service by identifying the backend access pods through Label selectors. Each Service corresponds to a ClusterIP address within a cluster. A cluster can access a Service through a ClusterIP address. To provide services outside the cluster, you can use NodePort or LoadBalancer.

Deployment. Yml configuration

The deployment.yml file is used to define deployment. Start by familiarizing yourself with the deployment configuration format with a simple deployment.yml configuration file.





In the figure above, deployment. yML is divided into 8 parts as follows:

  1. ApiVersion indicates the version of the current configuration format
  2. Kind specifies the resource type and, of course, Deployment
  3. Metadata is the metadata of the resource, where name is the required data item, and you can also specify label to label the resource
  4. The Spec section is the specification for this Deployment
  5. Spec. replicas specifies the number of Pod replicas
  6. Spec. Template defines basic Pod information, specified by spec.template. Metadata and spec.template
  7. Spec.template. metadata defines the metadata for the Pod. At least one label must be defined for the Service to identify the forward Pod. The label is specified in the key-value format
  8. Spec. Template. Spec describes the specification of the Pod. This section defines the properties of each container in the Pod. Name and image are required
In practice, there are more flexible and personalized configurations. We developed the relevant specifications in the deployment practice of Kubernetes, configured on the above infrastructure, and got the deployment.yml configuration file that met our actual needs.

In Kubernetes’ migration practice, we standardize the Deployment configuration mainly in the following aspects:

File templating

First, our deployment.yml configuration file is a template file with variables, as follows:

  1. apiVersion: apps/v1beta2
  2. kind: Deployment
  3. metadata:
  4. labels:
  5. app: __APP_NAME__
  6. group: __GROUP_NAME__
  7. name: __APP_NAME__
  8. namespace: __NAMESPACE__
Variables in the form of APPNAME__, __GROUPNAME and NAMESPACE are all replaced by variables corresponding to each GitLab project in CI/CD process. The goal is to have more projects using the same deployment.yml file so that it can be copied quickly and efficiently during Kubernetes migration.

The service name

  • {{groupName}}-{{projectname}} {groupname} -{{projectname}} Microservice – common. This name, named app_name, is used as the unique identity of each service in Kubernetes. These variables can be retrieved from gitlab-CI’s built-in variables without special configuration for each project.
  • The label used in Lables to identify the service remains the same as the Deployment name and is uniformly set to app:{{app_name}}.
The allocation of resources

The Node configuration policy is based on the project team as which nodes the PROJECT Pod runs on. The Pod of the project belonging to the same project team runs on the same batch of nodes. To do this, label each Node with a label of the form group:__GROUP_NAME__ and set the following parameters in the deployment.yml file to select the Node Node for Pod:

  1. .
  2. spec:
  3. .
  4. template:
  5. .
  6. spec:
  7. .
  8. affinity:
  9. nodeAffinity:
  10. requiredDuringSchedulingIgnoredDuringExecution:
  11. nodeSelectorTerms:
  12. – matchExpressions:
  13. – key: group
  14. operator: In
  15. values:
  16. – __GROUP_NAME__
  17. .
For some important online applications, limit is set to the same as Request. Kubernetes will ensure that these pods run properly when resources are insufficient. To improve resource utilization. For some non-core applications that do not occupy resources for a long time, the REQUEST of Pod can be appropriately reduced, so that Pod can be allocated to nodes with insufficient resources during scheduling to improve the utilization rate. However, if a node has insufficient resources, it will be expelled or killed by oom.

Liveness/Readiness configuration

Liveness is used to detect whether the container is alive. If the monitoring fails, the container restarts. Readiness determines whether to add to the forward list of services to receive request traffic by monitoring whether a container provides services properly. Readiness plays an important role in the upgrade process. During the upgrade process, an abnormal Pod of a new version is replaced by an old version. As a result, the entire application cannot provide services.

Each service must provide an accessible interface, and the corresponding monitoring and detection policy must be configured in the deployment.yml file.

  1. .
  2. spec:
  3. .
  4. template:
  5. .
  6. spec:
  7. .
  8. containers:
  9. – name: fpm
  10. livenessProbe:
  11. httpGet:
  12. path: /__PROJECT_NAME__
  13. port: 80
  14. initialDelaySeconds: 3
  15. periodSeconds: 5
  16. readinessProbe:
  17. httpGet:
  18. path: /__PROJECT_NAME__
  19. port: 80
  20. initialDelaySeconds: 3
  21. periodSeconds: 5
  22. .
  23. .
Upgrade Policy Configuration

Upgrade strategy We choose the RollingUpdate method, that is, during the upgrade process, gradually create a new version of Pod, and gradually kill the old version of Pod after the new version starts normally, and finally replace all the new version of Pod with the old version of Pod.





We can also set maxSurge and maxUnavailable values to control the maximum percentage of pods that can be added to the upgrade and the maximum percentage of pods that are unavailable during the upgrade, respectively.

The log configuration

Log-tail is used to collect container logs, and logs of all services are reported to a log-store of ali Cloud log service. The configuration in the deployment.yml file is as follows:

  1. .
  2. spec:
  3. .
  4. template:
  5. .
  6. spec:
  7. .
  8. containers:
  9. – name: fpm
  10. env:
  11. – name: aliyun_logs_vpgame
  12. value: stdout
  13. – name: aliyun_logs_vpgame_tags
  14. value: topic=__APP_NAME__
  15. .
  16. .
You can set environment variables to specify the Logstore to be uploaded and the corresponding tag. Name indicates the Logstore name. The logs of different services are distinguished by topic field.

Monitoring configuration

By adding Annotations to the Deployment, Prometheus can obtain the business monitoring data for each Pod. The configuration example is as follows:

  1. .
  2. spec:
  3. .
  4. template:
  5. metadata:
  6. annotations:
  7. prometheus.io/scrape: “true”
  8. prometheus.io/port: “80”
  9. prometheus.io/path: /{{ project_name }}/metrics
  10. .
IO /port indicates the port where the monitoring data is obtained, and Prometheus. IO /path indicates the path where the monitoring data is obtained.

Service. Yml The service. Yml configuration file describes services.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. annotations:
  5. service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
  6. labels:
  7. app: __APP_NAME__
  8. name: __APP_NAME__
  9. namespace: __NAMESPACE__
  10. spec:
  11. ports:
  12. – port: 80
  13. protocol: TCP
  14. targetPort: 80
  15. selector:
  16. app: __APP_NAME__
  17. Type: LoadBalancer Services can be considerably simpler than Deoloyment. The spec.ports parameters allow you to specify the exposed ports of the Service to be forwarded to the back-end Pod port. Spec. selector is the label that specifies the Pod to forward.
In addition, we provide the service through the LoadBalancer type, which is implemented by defining spec.type as LoadBalancer. By increasing the metadata. The annotations for service. The beta. Kubernetes. IO/alicloud – loadbalancer – address – type: An Intranet can create an ALI Cloud Intranet SLB when creating the Service as the entrance to request traffic to the Service.





As shown in the preceding figure, external-IP is the IP address of the SLB.

conclusion

On the basis of the above work, we divide each service into several categories (currently basically divided according to language), and then develop a unified CI/CD process for the services in each category through.gitlab-ci.yml. Similarly, Services in the same class share a Deployment and Service template. This allows us to quickly and efficiently migrate services to the Kubernetes environment.

Of course, this is only the first step on the way of migration practice, in Kubernetes service stability, performance, automatic scaling and other aspects need to be further explored and studied.

Author: Wu Chongbin

The original link

This article is the original content of the cloud habitat community, shall not be reproduced without permission.