A deployment kubelet

Kubelet runs on each worker node, receives requests sent by Kube-Apiserver, manages Pod containers, and executes interactive commands such as exec, RUN, logs, etc.

Kubelet automatically registers node information with Kube-Apiserver when it starts, and the built-in CAdvisor counts and monitors resource usage of nodes.

To ensure security, kubelet’s insecure HTTP port is closed during deployment, requests are authenticated and authorized, and unauthorized access (such as requests from Apiserver and Heapster) is denied.

1.1 get kubelet

Tip: The master01 node has downloaded the corresponding binary and can be directly distributed to the worker node.

1.2 distribute kubelet

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_ip in ${ALL_IPS[@]}
  4   do
  5     echo ">>> ${all_ip}"
  6     scp kubernetes/server/bin/kubelet root@${all_ip}:/opt/k8s/bin/
  7     ssh root@${all_ip} "chmod +x /opt/k8s/bin/*"
  8   done
Copy the code

Warning: You need to perform this step only on master01.

1.3 distribute kubeconfig

1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# for All_name in ${ALL_NAMES[@]} 4 do 5 echo ">>> ${all_name}" 6 7 # create token 8 export BOOTSTRAP_TOKEN=$(kubeadm token create  \ 9 --description kubelet-bootstrap-token \ 10 --groups system:bootstrappers:${all_name} \ 11 --kubeconfig ~/.kube/config) 13 kubectl config set-cluster kubernetes \ 15 --certificate-authority=/etc/kubernetes/cert/ca.pem \ 16 --embed-certs=true \ 17 --server=${KUBE_APISERVER} \ 18 --kubeconfig=kubelet-bootstrap-${all_name}. Kubeconfig 20 kubectl config set-credentials kubelet-bootstrap \ 22 --token=${BOOTSTRAP_TOKEN} \ 23 --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig 24 25 # Kubectl config set-context default \ 27 --cluster=kubernetes \ 28 --user=kubelet-bootstrap \ 29 kubectl config set-context default \ 27 --cluster=kubernetes \ 28 --user=kubelet-bootstrap \ 29 --kubeconfig=kubelet-bootstrap-${all_name}. Kubeconfig 30 31 kubectl config use-context default --kubeconfig=kubelet-bootstrap-${all_name}.kubeconfig 33 doneCopy the code

Explanation:

The token is written to KubeconFig. After bootstrap, Kube-Controller-Manager creates client and server certificates for Kubelet.

The token is valid for one day. After the token expires, it can no longer be used for Boostrap kubelet and will be cleaned by Kube-Controller-Manager tokencleaner.

After receiving Kubelet’s Bootstrap token, kube-Apiserver sets the user of the request to System :bootstrap:< token ID> and group to system:bootstrappers, A ClusterRoleBinding will then be set for this group.

1 [root@master01 work]# kubeadm token list --kubeconfig ~/. Kube /config # check kubeadm token 2 [root@master01 work]# Kubectl get secrets - n kube - system | grep bootstrap - token Secret # to check each token associationsCopy the code

Warning: You need to perform this step only on master01.

1.4 Distributing Bootstrap kubeconfig

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_name in ${ALL_NAMES[@]}
  4   do
  5     echo ">>> ${all_name}"
  6     scp kubelet-bootstrap-${all_name}.kubeconfig root@${all_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
  7   done
Copy the code

Warning: You need to perform this step only on master01.

1.5 Creating the Kubelet Parameter Profile

Starting from V1.10, some kubelet parameters need to be configured in the configuration file. It is recommended to create a Kubelet configuration file.

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 
  4 [root@master01 work]# cat > kubelet-config.yaml.template <<EOF
  5 kind: KubeletConfiguration
  6 apiVersion: kubelet.config.k8s.io/v1beta1
  7 address: "##ALL_IP##"
  8 staticPodPath: ""
  9 syncFrequency: 1m
 10 fileCheckFrequency: 20s
 11 httpCheckFrequency: 20s
 12 staticPodURL: ""
 13 port: 10250
 14 readOnlyPort: 0
 15 rotateCertificates: true
 16 serverTLSBootstrap: true
 17 authentication:
 18   anonymous:
 19     enabled: false
 20   webhook:
 21     enabled: true
 22   x509:
 23     clientCAFile: "/etc/kubernetes/cert/ca.pem"
 24 authorization:
 25   mode: Webhook
 26 registryPullQPS: 0
 27 registryBurst: 20
 28 eventRecordQPS: 0
 29 eventBurst: 20
 30 enableDebuggingHandlers: true
 31 enableContentionProfiling: true
 32 healthzPort: 10248
 33 healthzBindAddress: "##ALL_IP##"
 34 clusterDomain: "${CLUSTER_DNS_DOMAIN}"
 35 clusterDNS:
 36   - "${CLUSTER_DNS_SVC_IP}"
 37 nodeStatusUpdateFrequency: 10s
 38 nodeStatusReportFrequency: 1m
 39 imageMinimumGCAge: 2m
 40 imageGCHighThresholdPercent: 85
 41 imageGCLowThresholdPercent: 80
 42 volumeStatsAggPeriod: 1m
 43 kubeletCgroups: ""
 44 systemCgroups: ""
 45 cgroupRoot: ""
 46 cgroupsPerQOS: true
 47 cgroupDriver: cgroupfs
 48 runtimeRequestTimeout: 10m
 49 hairpinMode: promiscuous-bridge
 50 maxPods: 220
 51 podCIDR: "${CLUSTER_CIDR}"
 52 podPidsLimit: -1
 53 resolvConf: /etc/resolv.conf
 54 maxOpenFiles: 1000000
 55 kubeAPIQPS: 1000
 56 kubeAPIBurst: 2000
 57 serializeImagePulls: false
 58 evictionHard:
 59   memory.available:  "100Mi"
 60 nodefs.available:  "10%"
 61 nodefs.inodesFree: "5%"
 62 imagefs.available: "15%"
 63 evictionSoft: {}
 64 enableControllerAttachDetach: true
 65 failSwapOn: true
 66 containerLogMaxSize: 20Mi
 67 containerLogMaxFiles: 10
 68 systemReserved: {}
 69 kubeReserved: {}
 70 systemReservedCgroup: ""
 71 kubeReservedCgroup: ""
 72 enforceNodeAllocatable: ["pods"]
 73 EOF
Copy the code

Warning: You need to perform this step only on master01.

1.6 Distributing kubelet parameter configuration files

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_ip in ${ALL_IPS[@]}
  4   do
  5     echo ">>> ${all_ip}"
  6     sed -e "s/##ALL_IP##/${all_ip}/" kubelet-config.yaml.template > kubelet-config-${all_ip}.yaml.template
  7     scp kubelet-config-${all_ip}.yaml.template root@${all_ip}:/etc/kubernetes/kubelet-config.yaml
  8   done
Copy the code

Warning: You need to perform this step only on master01.

1.7 Creating Kubelet Systemd

1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# cat > kubelet.service.template <<EOF 4 [Unit] 5 Description=Kubernetes Kubelet 6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes 7 After=docker.service 8 Requires=docker.service 9 10 [Service] 11 WorkingDirectory=${K8S_DIR}/kubelet 12 ExecStart=/opt/k8s/bin/kubelet \\ 13 --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\ 14 --cert-dir=/etc/kubernetes/cert \\ 15 --cgroup-driver=cgroupfs \\ 16 --cni-conf-dir=/etc/cni/net.d \\ 17 --container-runtime=docker \\ 18 --container-runtime-endpoint=unix:///var/run/dockershim.sock \\ 19 --root-dir=${K8S_DIR}/kubelet \\ 20 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\ 21 --config=/etc/kubernetes/kubelet-config.yaml \\ 22 --hostname-override=##ALL_NAME## \\ 23 --pod-infra-container-image=k8s.gcr. IO /pause-amd64:3.2 \\ 24 --image-pull-progress-deadline=15m \\ 25 --volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\ 26 --logtostderr=true \\ 27 --v=2 28 Restart=always 29 RestartSec=5 30 StartLimitInterval=0 31 32 [Install] 33 WantedBy=multi-user.target 34 EOFCopy the code

Warning: You need to perform this step only on master01.

Explanation:

  • If the –hostname-override option is set, kube-proxy also needs to set this option, otherwise the Node will not be found.
  • –bootstrap-kubeconfig: points to the bootstrap kubeconfig file in which kubelet sends a TLS Bootstrapping request to kube-Apiserver using the user name and token;
  • K8S creates the certificate and private key file in the –cert-dir directory after approving kubelet’s CSR request, and then writes the –kubeconfig file;
  • –pod-infra-container-image does not use redhat’s pod-infrastructure:latest image, which cannot reclaim container zombies.

1.8 Distribute Kubelet Systemd

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_name in ${ALL_NAMES[@]}
  4   do
  5     echo ">>> ${all_name}"
  6     sed -e "s/##ALL_NAME##/${all_name}/" kubelet.service.template > kubelet-${all_name}.service
  7     scp kubelet-${all_name}.service root@${all_name}:/etc/systemd/system/kubelet.service
  8   done
Copy the code

Warning: You need to perform this step only on master01.

Two Startup Verification

2.1 license

Kubelet starts with kubeletConfig. If not, use the kubeconfig file specified by –bootstrap-kubeconfig to send a certificate signing request (CSR) to kube-Apiserver.

After receiving the CSR request, kube-Apiserver authenticates the tokens in it. After passing the authentication, the user of the request is set to system:bootstrap:<Token ID>, and the group is set to system:bootstrappers, This process is called Bootstrap Token Auth.

By default, the user and group do not have permission to create CSRS, so kubelet will fail to start. Create a ClusterRoleBinding as follows: Bind Group System :bootstrappers with ClusterRole System :node-bootstrapper.

  1 [root@master01 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
Copy the code

Warning: You need to perform this step only on master01.

2.2 start kubelet

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_name in ${ALL_NAMES[@]}
  4   do
  5     echo ">>> ${all_name}"
  6     ssh root@${all_name} "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
  7     ssh root@${all_name} "/usr/sbin/swapoff -a"
  8     ssh root@${all_name} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
  9   done
Copy the code

After kubelet is started, use –bootstrap-kubeconfig to send a CSR request to kube-apiserver. When the CSR is approved, Kube-controller-manager creates TLS client certificates, private keys, and –kubeletconfig files for kubelet.

Note: kube-controller-manager requires the –cluster-signing-cert-file and –cluster-signing-key-file parameters to create the certificate and private key for TLS Bootstrap.

Tip: You must create a working directory before starting the service.

Disable swap partition or Kubelet will fail to start.

Warning: You need to perform this step only on master01.

2.3 Viewing the Kubelet Service

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for all_name in ${ALL_NAMES[@]}
  4   do
  5     echo ">>> ${all_name}"
  6     ssh root@${all_name} "systemctl status kubelet"
  7   done
  8 [root@master01 work]# kubectl get csr
  9 [root@master01 work]# kubectl get nodes
Copy the code

Warning: You need to perform this step only on master01.

Approve a CSR request

3.1 Automatically approve a CSR request

Create three ClusterRoleBinding for automatically approving Client, Renew Client, and Renew Server certificates, respectively.

1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# cat > csr-crb.yaml <<EOF 4 # Approve all CSRs for the group "system:bootstrappers" 5 kind: ClusterRoleBinding 6 apiVersion: rbac.authorization.k8s.io/v1 7 metadata: 8 name: auto-approve-csrs-for-group 9 subjects: 10 - kind: Group 11 name: system:bootstrappers 12 apiGroup: rbac.authorization.k8s.io 13 roleRef: 14 kind: ClusterRole 15 name: system:certificates.k8s.io:certificatesigningrequests:nodeclient 16 apiGroup: rbac.authorization.k8s.io 17 --- 18 # To let a node of the group "system:nodes" renew its own credentials 19 kind: ClusterRoleBinding 20 apiVersion: rbac.authorization.k8s.io/v1 21 metadata: 22 name: node-client-cert-renewal 23 subjects: 24 - kind: Group 25 name: system:nodes 26 apiGroup: rbac.authorization.k8s.io 27 roleRef: 28 kind: ClusterRole 29 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 30 apiGroup: rbac.authorization.k8s.io 31 --- 32 # A ClusterRole which instructs the CSR approver to approve a node requesting a 33 #  serving cert matching its client cert. 34 kind: ClusterRole 35 apiVersion: rbac.authorization.k8s.io/v1 36 metadata: 37 name: approve-node-server-renewal-csr 38 rules: 39 - apiGroups: ["certificates.k8s.io"] 40 resources: ["certificatesigningrequests/selfnodeserver"] 41 verbs: ["create"] 42 --- 43 # To let a node of the group "system:nodes" renew its own server credentials 44 kind: ClusterRoleBinding 45 apiVersion: rbac.authorization.k8s.io/v1 46 metadata: 47 name: node-server-cert-renewal 48 subjects: 49 - kind: Group 50 name: system:nodes 51 apiGroup: rbac.authorization.k8s.io 52 roleRef: 53 kind: ClusterRole 54 name: approve-node-server-renewal-csr 55 apiGroup: rbac.authorization.k8s.io 56 EOF 57 [root@master01 work]# kubectl apply -f csr-crb.yamlCopy the code

Explanation:

Auto-approve -csrs-for-group: automatically approve the first CSR for a node; Note that the Group requested for the first CSR is System :bootstrappers;

Node-client-cert-renewal: automatically approve a node’s subsequent expired client certificate. The certificate Group is System: Nodes.

Node-server-cert-renewal: automatically approve the subsequent expired server certificate of a node. The certificate Group is System: Nodes.

Warning: You need to perform this step only on master01.

3.2 Check kubelet

1 [root @ master01 work] # kubectl get CSR | grep boot # wait for a period of time (1-10 minutes), CSR is automatically approved for all three nodes. 2 [root@master01 work]# kubectl get nodes # All nodes are readyCopy the code

  1 [root@master01 ~]# ls -l /etc/kubernetes/kubelet.kubeconfig
  2 [root@master01 ~]# ls -l /etc/kubernetes/cert/ | grep kubelet
Copy the code

Warning: You need to perform this step only on master01.

3.3 Manually Approve a Server CERT CSR

For security reasons, CSR Approving controllers will not automatically approve kubelet Server certificate signing requests, you need to approve them manually.

  1 [root@master01 ~]# kubectl get csr | grep node
Copy the code

  1 [root@master01 ~]# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
  2 [root@master01 ~]# ls -l /etc/kubernetes/cert/kubelet-*
Copy the code

Warning: You need to perform this step only on master01.

Four Kubelet API interface

4.1 API interface provided by Kubelet

1 / root @ master01 ~ # sudo netstat LNPT | grep kubelet # check kubelet listener portCopy the code

Explanation:

  • 10248: HEALTHZ HTTP service;
  • 10250: HTTPS service that requires authentication and authorization to access this port (even if accessing/HealthZ);
  • Read-only port 10255 is disabled.
  • Starting with K8S V1.10, the — cAdvisor-port parameter (default port 4194) is removed, and access to CAdvisor UI & API is not supported.

Warning: You need to perform this step only on master01.

4.2 Kubelet API authentication and authorization

Kubelet has configured the following authentication parameters:

  • Authentication. Anonymous. Enabled: set to false, does not allow anonymous access port 10250;
  • Authentication. X509. ClientCAFile: specify the signature client certificate of the CA certificate, open the HTTPs certificate authentication;
  • Authentication. Webhook. Enabled = true: open HTTPs bearer token authentication.

The following authorization parameters are configured:

Authroization. mode=Webhook: Enable RBAC authorization.

Upon receiving the request, Kubelet uses clientCAFile to authenticate the certificate signatures or to look up whether or not bearer tokens are valid. If both fail, the request is rejected with an Unauthorized message.

1 / root @ master01 ~ # curl - s -- cacert/etc/kubernetes/cert/ca. Pem 2 Unauthorized 3 at https://172.24.8.71:10250/metrics [root@master01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: 4 Unauthorized Bearer 123456 "https://172.24.8.71:10250/metricsCopy the code

If the certificate is authenticated, Kubelet uses SubjectAccessReview API to send a request to Kube-Apiserver to query whether the user or group corresponding to the certificate or token has the permission to operate resources (RBAC).

Warning: You need to perform this step only on master01.

4.3 Certificate Authentication and Authorization

1 [root@master01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.24.8.71:10250/metrics # default permissions less than two Forbidden (user = system: kube - controller - manager, verb = get, resource = nodes, subresource=metrics) 3 [root@master01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem - key/opt/k8s/work/admin - key. Pem admin # https://172.24.8.71:10250/metrics | head using the supreme authorityCopy the code

Explanation:

The values of –cacert, –cert, and –key must be the file path./admin.pem cannot be omitted.

Warning: You need to perform this step only on master01.

4.4 Creating BEAR Token Authentication and Authorization

  1 [root@master01 ~]# kubectl create sa kubelet-api-test
  2 [root@master01 ~]# kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
  3 [root@master01 ~]# SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
  4 [root@master01 ~]# TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
  5 [root@master01 ~]# echo ${TOKEN}
Copy the code

  1 [root@master01 ~]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.24.8.71:10250/metrics | head
Copy the code

Warning: You need to perform this step only on master01.