This is the first day of my participation in the More text Challenge. For details, see more text Challenge

A background

In most Internet companies, the load of application of the distribution of peak valley, there are certain rules to follow, the order application, for example, at 12 noon, at 6 PM and 9 PM show obvious peak flow, its applied load, also the highest of the type of business more long time monitoring data, and business users, It can accurately predict the peak of application. Generally, there is a 3-4 times difference between the peak and the trough of application resources, and the peak may come not slowly rising, but instantly loaded.

In the cloud native, the presence of containers allows for more standard, lightweight, and automatic dynamic capacity expansion. In Kubernetes, the number of replicase can be simply restored. For elastic capacity expansion, the HPA can be defined to dynamically adjust the number of replicas according to different metric. Due to the pull delay of back-end POD in HPA, it is obvious that the user needs cannot be well met in this scenario. However, if only manually expanding POD in advance, it is obviously not reasonable for such periodic business.

Two schemes

2.1 train of thought

Standard HPA is scaled based on the threshold of indicators. Common indicators are CPU and memory. Of course, customized indicators such as QPS and connection number can also be scaled. However, there is a problem: resource-based scaling has a certain delay, which mainly includes the collection delay (minute), judgment delay (minute), and scaling delay (minute). In the figure above, we can find that the peak burr of the load is still very sharp, which may cause that the number of loads cannot be changed in time due to the scaling delay at the level of HPA minutes. In a short time, the overall load of the application is skyrocketing and the response time is slow. Especially for some game businesses, the business shake caused by high load will cause a very poor experience for players.

To address this scenario, Ali currently provides a Kube-cronHpa-Controller that specializes in scenarios where resource profiles are periodic. Developers can define the time schedule according to the periodicity of resource portrait, expand resources in advance, and recycle resources when the trough comes. The bottom layer is combined with the node scaling capability of cluster-AutoScaler to provide resource cost savings.

2.2 cronhpa

Cronhpa, which leverages the underlying use of go-Cron, supports more rules.

  Field name   | Mandatory? | Allowed values  | Allowed special characters
  ----------   | ---------- | --------------  | --------------------------
  Seconds      | Yes        | 0-59            | * / , -
  Minutes      | Yes        | 0-59            | * / , -
  Hours        | Yes        | 0-23            | * / , -
  Day of month | Yes        | 1-31            | * / , - ?
  Month        | Yes        | 1-12 or JAN-DEC | * / , -
  Day of week  | Yes        | 0-6 or SUN-SAT  | * / , - ?    
Copy the code

kubernetes-cronhpa-controller is a kubernetes cron horizontal pod autoscaler controller using crontab like scheme. You can use CronHorizontalPodAutoscaler with any kind object defined in kubernetes which support scale subresource(such as Deployment and StatefulSet).

Iii Installation and Deployment

3.1 Downloading Related Resources

git clone https://github.com/AliyunContainerService/kubernetes-cronhpa-controller.git
Copy the code

3.2 installation

install CRD

kubectl apply -f config/crds/autoscaling_v1beta1_cronhorizontalpodautoscaler.yaml
Copy the code

install RBAC settings

# create ClusterRole 
kubectl apply -f config/rbac/rbac_role.yaml

# create ClusterRolebinding and ServiceAccount 
kubectl apply -f config/rbac/rbac_role_binding.yaml

[root@master ~]# kubectl api-resources |grep cronhpa
cronhorizontalpodautoscalers      cronhpa      autoscaling.alibabacloud.com   true         CronHorizontalPodAutoscaler
Copy the code

deploy kubernetes-cronhpa-controller

kubectl apply -f config/deploy/deploy.yaml
Copy the code

verify installation

Those who qualify can qualify as university. Kubectl get deploy kubernetes-cronhpa-controller-n kube-system-o wide ➜ kubernetes-cronhpa-controller git:(master) qualify kubectl get deploy kubernetes-cronhpa-controller -n kube-system NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kubernetes-cronhpa-controller 1 1 1 1 49sCopy the code

Four test

Go to the Examples Folder directory

Deploy to create workload and cronHpA

kubectl apply -f examples/deployment_cronhpa.yaml 
Copy the code

View the number of deploy copies

Those who qualify can qualify kubectl get deploy nginx-deployment-basic ➜ kubernetes-cronhpa-controller git:(master) qualify kubectl get deploy nginx-deployment-basic NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment-basic 2 2 2 2 9sCopy the code

Check the cronhpa status

[root@master kubernetes-cronhpa-controller]# kubectl describe cronhpa cronhpa-sample 'Name: cronhpa-sample Namespace: The default Labels: the controller - tools. K8s. IO = 1.0 Annotations: kubectl. Kubernetes. IO/last - applied - configuration: {"apiVersion":"autoscaling.alibabacloud.com/v1beta1","kind":"CronHorizontalPodAutoscaler","metadata":{"annotations":{}," labels":{"controll... API Version: autoscaling.alibabacloud.com/v1beta1 Kind: CronHorizontalPodAutoscaler Metadata: Creation Timestamp: 2020-07-31T07:27:12Z Generation: 5 Resource Version: 67753389 Self Link: /apis/autoscaling.alibabacloud.com/v1beta1/namespaces/default/cronhorizontalpodautoscalers/cronhpa-sample UID: 3be95264-9b08-4efc-b9f9-db1aca7a80d8 Spec: Exclude Dates: <nil> Jobs: Name: scale-down Run Once: false Schedule: 30 */1 * * * * Target Size: 1 Name: scale-up Run Once: false Schedule: 0 */1 * * * * Target Size: 3 Scale Target Ref: API Version: apps/v1beta2 Kind: Deployment Name: nginx-deployment-basic Status: Conditions: Job Id: 184b967e-a149-488d-b4ae-765f9a792f93 Last Probe Time: 2020-07-31T07:28:31Z Message: cron hpa job scale-down executed successfully. current replicas:3, desired replicas:1. Name: scale-down Run Once: false Schedule: 30 */1 * * * * State: Succeed Target Size: 1 Job Id: 870b014f-cfbc-4f60-a628-16f2995c288d Last Probe Time: 2020-07-31T07:28:01Z Message: cron hpa job scale-up executed successfully. current replicas:1, desired replicas:3. Name: scale-up Run Once: false Schedule: 0 */1 * * * * State: Succeed Target Size: 3 Exclude Dates: <nil> Scale Target Ref: API Version: apps/v1beta2 Kind: Deployment Name: nginx-deployment-basic Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Succeed 87s cron-horizontal-pod-autoscaler cron hpa job scale-down executed successfully. current replicas:2, desired replicas:1. Normal Succeed 57s cron-horizontal-pod-autoscaler cron hpa job scale-up executed successfully. current replicas:1, desired replicas:3. Normal Succeed 27s cron-horizontal-pod-autoscaler cron hpa job scale-down executed successfully. current replicas:3, desired replicas:1.Copy the code

In this example, the set is to expand to 3 pods at 0 seconds per minute and shrink to 1 Pod at 30 seconds per minute. If the execution is normal, we can see two changes in the number of loads within 30 seconds. By checking the increase and reduction of POD, we can more easily control the number of POD regularly, so as to maximize the efficiency of resources in band business.

Five reflection

At present, stateless applications can be combined with HPA or CronHPA to cope with the pressure of horizontal scaling. For stateful applications, Vertical Pod Autoscaler (VPA) can be used for implementation. According to the business user profile, the capacity level can be accurately estimated and the cost can be minimized to automatically meet the business requirements.

Refer to the link

www.lagou.com/lgeduarticl…
Github.com/AliyunConta…

Real-time elastic scaling CronHPA in cloud native