Background: I feel that K8S does not have many resources, but the total request ratio of K8S is very high, so I conduct investigation

The screening process

  • From the above figure, the total CPU limit of the cluster is extremely high and the cluster usage is extremely low. Note There is room for reducing the request value.

  • From the perspective of namespace, production space is obviously abnormal. The average request value of a POD is 1, and the limit value is 2500

  • There are 1000 pods. No wonder the request and Limit values are so large

  • More than 1000 pods are generated by cronjobs, because the namespace is configured with isTIO automatic injection. When the container in cronJobs finishes running, the ISTIo-proxy container still runs normally, so the pod does not exit. Github.com/istio/istio… Found the solution in this issue, it is in a cronjob (cronjob. Spec. JobTemplate. Spec. Metadata) metadata annotation sidecars. Add istio. IO/inject: “False”, and write a script to exit the existing IStio-proxy one by one. The curl – Max – time 2 – s – f – XPOST http://127.0.0.1:15000/quitquitquit

kubectl exec -n production -t pod_name -c istio-proxy -- curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit
Copy the code

The results of

  • Now the cluster limit and request look much more normal, but the problem of low utilization still exists, and we will optimize it later