Get Kubernetes binary file
1.1 Master Node Services
The Kubernetes Master node runs the following components:
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kube-nginx
Kube-apiserver, kube-Scheduler, and kube-Controller-Manager all run in multi-instance mode:
Kube-scheduler and Kube-Controller-Manager automatically elect a leader instance, while other instances are in blocking mode. When the leader dies, a new leader is elected to ensure service availability.
Kube-apiserver is stateless and requires proxy access via kube-nginx to ensure service availability.
1.2 installation Kubernetes
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# wget https://storage.googleapis.com/kubernetes-release/release/v1.18.3/kubernetes-server-linux-amd64.tar.gz 3 [root @ master01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz 4 [root@master01 work]# cd kubernetes 5 [root@master01 kubernetes]# tar -xzvf kubernetes-src.tar.gzCopy the codeWarning: You need to perform this step only on master01.
1.3 distribute Kubernetes
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp -rp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm, kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/ 7 ssh root@${master_ip} "chmod +x /opt/k8s/bin/*" 8 doneCopy the codeWarning: You need to perform this step only on master01.
Create kube-apiserver certificate
2.1 Introduction to the Ha Apiserver
This experiment deploates the steps of a three-instance Kube-Apiserver cluster with proxy access via kube-nginx to ensure service availability.
2.2 Creating the Kubernetes Certificate and Private Key
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# cat > kubernetes-csr.json <<EOF 4 { 5 "CN": "kubernetes", 6 "hosts": [7 "127.0.0.1", 8 "172.24.8.71", 9 "172.24.8.72", 10 "172.24.8.73," 11 "172.24.8.100", 12 "${CLUSTER_KUBERNETES_SVC_IP}", 13 "kubernetes", 14 "kubernetes.default", 15 "kubernetes.default.svc", 16 "kubernetes.default.svc.cluster", 17 "kubernetes.default.svc.cluster.local." 18 ], 19 "key": { 20 "algo": "rsa", 21 "size": 2048 22 }, 23 "names": [ 24 { 25 "C": "CN", 26 "ST": "Shanghai", 27 "L": "Shanghai", 28 "O": "K8s ", 29 "OU": "System" 30} 31] 32} 33 EOF # create Kubernetes CA certificate request fileCopy the codeWarning: You need to perform this step only on master01.
Explanation:
The hosts field specifies the IP addresses and domain names authorized to use the certificate. The IP addresses of the master node and the IP addresses and domain names of the Kubernetes service are listed.
The kubernetes service IP address is automatically created by apiserver. It is usually the first IP address in the network segment specified by the –service-cluster-ip-range parameter. You can run the following command to obtain the IP address:
# kubectl get svc kubernetes
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kubernetes-csr.json | Cfssljson -bare kubernetes # generate key and certificateCopy the codeWarning: You need to perform this step only on master01.
2.3 Distributing certificates and Private Keys
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert"
7 scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/
8 done
Copy the codeWarning: You need to perform this step only on master01.
Configure kube-Apiserver audit
3.1 Creating an Encryption Profile
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# cat > encryption-config.yaml <<EOF
4 kind: EncryptionConfig
5 apiVersion: v1
6 resources:
7 - resources:
8 - secrets
9 providers:
10 - aescbc:
11 keys:
12 - name: key1
13 secret: ${ENCRYPTION_KEY}
14 - identity: {}
15 EOF
Copy the codeWarning: You need to perform this step only on master01.
3.2 Distributing an encryption Configuration File
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/
7 done
Copy the codeWarning: You need to perform this step only on master01.
3.3 Creating an Audit Policy File
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# cat > audit-policy.yaml <<EOF
4 apiVersion: audit.k8s.io/v1beta1
5 kind: Policy
6 rules:
7 # The following requests were manually identified as high-volume and low-risk, so drop them.
8 - level: None
9 resources:
10 - group: ""
11 resources:
12 - endpoints
13 - services
14 - services/status
15 users:
16 - 'system:kube-proxy'
17 verbs:
18 - watch
19
20 - level: None
21 resources:
22 - group: ""
23 resources:
24 - nodes
25 - nodes/status
26 userGroups:
27 - 'system:nodes'
28 verbs:
29 - get
30
31 - level: None
32 namespaces:
33 - kube-system
34 resources:
35 - group: ""
36 resources:
37 - endpoints
38 users:
39 - 'system:kube-controller-manager'
40 - 'system:kube-scheduler'
41 - 'system:serviceaccount:kube-system:endpoint-controller'
42 verbs:
43 - get
44 - update
45
46 - level: None
47 resources:
48 - group: ""
49 resources:
50 - namespaces
51 - namespaces/status
52 - namespaces/finalize
53 users:
54 - 'system:apiserver'
55 verbs:
56 - get
57
58 # Don't log HPA fetching metrics.
59 - level: None
60 resources:
61 - group: metrics.k8s.io
62 users:
63 - 'system:kube-controller-manager'
64 verbs:
65 - get
66 - list
67
68 # Don't log these read-only URLs.
69 - level: None
70 nonResourceURLs:
71 - '/healthz*'
72 - /version
73 - '/swagger*'
74
75 # Don't log events requests.
76 - level: None
77 resources:
78 - group: ""
79 resources:
80 - events
81
82 # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
83 - level: Request
84 omitStages:
85 - RequestReceived
86 resources:
87 - group: ""
88 resources:
89 - nodes/status
90 - pods/status
91 users:
92 - kubelet
93 - 'system:node-problem-detector'
94 - 'system:serviceaccount:kube-system:node-problem-detector'
95 verbs:
96 - update
97 - patch
98
99 - level: Request
100 omitStages:
101 - RequestReceived
102 resources:
103 - group: ""
104 resources:
105 - nodes/status
106 - pods/status
107 userGroups:
108 - 'system:nodes'
109 verbs:
110 - update
111 - patch
112
113 # deletecollection calls can be large, don't log responses for expected namespace deletions
114 - level: Request
115 omitStages:
116 - RequestReceived
117 users:
118 - 'system:serviceaccount:kube-system:namespace-controller'
119 verbs:
120 - deletecollection
121
122 # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
123 # so only log at the Metadata level.
124 - level: Metadata
125 omitStages:
126 - RequestReceived
127 resources:
128 - group: ""
129 resources:
130 - secrets
131 - configmaps
132 - group: authentication.k8s.io
133 resources:
134 - tokenreviews
135 # Get repsonses can be large; skip them.
136 - level: Request
137 omitStages:
138 - RequestReceived
139 resources:
140 - group: ""
141 - group: admissionregistration.k8s.io
142 - group: apiextensions.k8s.io
143 - group: apiregistration.k8s.io
144 - group: apps
145 - group: authentication.k8s.io
146 - group: authorization.k8s.io
147 - group: autoscaling
148 - group: batch
149 - group: certificates.k8s.io
150 - group: extensions
151 - group: metrics.k8s.io
152 - group: networking.k8s.io
153 - group: policy
154 - group: rbac.authorization.k8s.io
155 - group: scheduling.k8s.io
156 - group: settings.k8s.io
157 - group: storage.k8s.io
158 verbs:
159 - get
160 - list
161 - watch
162
163 # Default level for known APIs
164 - level: RequestResponse
165 omitStages:
166 - RequestReceived
167 resources:
168 - group: ""
169 - group: admissionregistration.k8s.io
170 - group: apiextensions.k8s.io
171 - group: apiregistration.k8s.io
172 - group: apps
173 - group: authentication.k8s.io
174 - group: authorization.k8s.io
175 - group: autoscaling
176 - group: batch
177 - group: certificates.k8s.io
178 - group: extensions
179 - group: metrics.k8s.io
180 - group: networking.k8s.io
181 - group: policy
182 - group: rbac.authorization.k8s.io
183 - group: scheduling.k8s.io
184 - group: settings.k8s.io
185 - group: storage.k8s.io
186
187 # Default level for all other requests.
188 - level: Metadata
189 omitStages:
190 - RequestReceived
191 EOF
Copy the codeWarning: You need to perform this step only on master01.
3.4 Distributing policy Files
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml
7 done
Copy the codeNote: the audit knowledge reference: www.cnblogs.com/liabio/p/11…
Warning: You need to perform this step only on master01.
Metrics – four configuration server
4.1 Creating a certificate and key for accessing metrics-server
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# cat > proxy-client-csr.json <<EOF 3 { 4 "CN": "system:metrics-server", 5 "hosts": [], 6 "key": { 7 "algo": "rsa", 8 "size": 2048 9 }, 10 "names": [ 11 { 12 "C": "CN", 13 "ST": "Shanghai", 14 "L": "Shanghai", 15 "O": "k8s", 16 "OU": "System" 17} 18] 19} 20 EOF # create metrics-server CA certificate request fileCopy the codeWarning: You need to perform this step only on master01.
Explanation:
CN must be in the — requesTheader-allowed-names parameter of kube-apiserver; otherwise, insufficient permissions will be displayed when you access metrics.
1 [root@master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 2 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 3 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client Generate keys and certificatesCopy the codeWarning: You need to perform this step only on master01.
4.2 Distributing certificates and Private Keys
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/
7 done
Copy the codeWarning: You need to perform this step only on master01.
Kube – apiserver five configuration
5.1 Creating kube-apiserver systemd
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# cat > kube-apiserver.service.template <<EOF 4 [Unit] 5 Description=Kubernetes API Server 6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes 7 After=network.target 8 9 [Service] 10 WorkingDirectory=${K8S_DIR}/kube-apiserver 11 ExecStart=/opt/k8s/bin/kube-apiserver \\ 12 --insecure-port=0 \\ 13 --secure-port=6443 \\ 14 --bind-address=##MASTER_IP## \\ 15 --advertise-address=##MASTER_IP## \\ 16 --default-not-ready-toleration-seconds=360 \\ 17 --default-unreachable-toleration-seconds=360 \\ 18 --feature-gates=DynamicAuditing=true \\ 19 --max-mutating-requests-inflight=2000 \\ 20 --max-requests-inflight=4000 \\ 21 --default-watch-cache-size=200 \\ 22 --delete-collection-workers=2 \\ 23 --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\ 24 --etcd-cafile=/etc/kubernetes/cert/ca.pem \\ 25 --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\ 26 --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\ 27 --etcd-servers=${ETCD_ENDPOINTS} \\ 28 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\ 29 --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\ 30 --audit-dynamic-configuration \\ 31 --audit-log-maxage=30 \\ 32 --audit-log-maxbackup=3 \\ 33 --audit-log-maxsize=100 \\ 34 --audit-log-truncate-enabled=true \\ 35 --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\ 36 --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\ 37 --profiling \\ 38 --anonymous-auth=false \\ 39 --client-ca-file=/etc/kubernetes/cert/ca.pem \\ 40 --enable-bootstrap-token-auth=true \\ 41 --requestheader-allowed-names="system:metrics-server" \\ 42 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\ 43 --requestheader-extra-headers-prefix=X-Remote-Extra- \\ 44 --requestheader-group-headers=X-Remote-Group \\ 45 --requestheader-username-headers=X-Remote-User \\ 46 --service-account-key-file=/etc/kubernetes/cert/ca.pem \\ 47 --authorization-mode=Node,RBAC \\ 48 --runtime-config=api/all=true \\ 49 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Mu tatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \\ 50 --allow-privileged=true \\ 51 --apiserver-count=3 \\ 52 --event-ttl=168h \\ 53 --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\ 54 --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\ 55 --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\ 56 --kubelet-https=true \\ 57 --kubelet-timeout=10s \\ 58 --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\ 59 --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\ 60 --service-cluster-ip-range=${SERVICE_CIDR} \\ 61 --service-node-port-range=${NODE_PORT_RANGE} \\ 62 --logtostderr=true \\ 63 --v=2 64 Restart=on-failure 65 RestartSec=10 66 Type=notify 67 LimitNOFILE=65536 68 69 [Install] 70 WantedBy=multi-user.target 71 EOFCopy the codeWarning: You need to perform this step only on master01.
Pass the file containing the policy to kube-Apiserver using the –audit-policy-file flag. If this flag is not set, events are not logged.
Explanation:
- –advertise-address: IP address advertised by apiserver (back-end IP address of Kubernetes service);
- –default-*-toleration-seconds: Set thresholds for node anomalies;
- — Max -*-requests-inflight: specifies the maximum threshold for requests.
- –etcd-* : certificate to access etCD and etCD server address;
- –experimental-encryption-provider-config: specifies the configuration used to encrypt secret in etCD;
- –bind-address: specifies the IP address that HTTPS listens to. It cannot be 127.0.0.1. Otherwise, external users cannot access its security port 6443.
- –secret-port: HTTPS listening port;
- –insecure-port=0: disable listening on HTTP insecure port (8080);
- — TLS -*-file: specifies the certificate, private key, and CA file used by apiserver.
- –audit-* : specifies parameters related to audit policies and audit log files.
- –client-ca-file: verifies the certificate of the client (kue-controller-manager, KuBE-Scheduler, kubelet, kube-proxy, etc.) request.
- –enable-bootstrap-token-auth: enables kubelet Bootstrap token authentication.
- –requestheader-* : kube-apiserver aggregator layer configuration parameters, proxy-client & HPA need to use;
- –requestheader-client-ca-file: used to sign certificates specified by –proxy-client-cert-file and –proxy-client-key-file. Used when the metric Aggregator is enabled;
- –requestheader-allowed-names: cannot be empty and must be separated by commas. –proxy-client-cert-file SPECIFIES the CN name of the certificate. Set this parameter to aggregator.
- – service – account – key – file: The public key file of the ServiceAccount Token is used. The –service-account-private-key-file of kube-controller-Manager specifies the private key file. The two files can be used together.
- — Run-time config= API /all=true: enable APIs of all versions, such as autoscaling/v2alpha1;
- –authorization-mode=Node,RBAC, –anonymous-auth=false: Enable the Node and RBAC authorization modes and reject unauthorized requests.
- Enable-admission-plugins: enable some plugins that are disabled by default;
- –allow-privileged: run the privileged container.
- –apiserver-count=3: specifies the number of instances of apiserver.
- –event-ttl: specifies the time to save events.
- –kubelet-* : if specified, use HTTPS to access the kubelet APIs; You need to define RBAC rules for the users corresponding to the certificate (kubernetes*.pem user is kubernetes); otherwise, an unauthorized message is displayed when accessing the Kubelet API.
- –proxy-client-* : proxy used by apiserver to access metrics-server;
- –service-cluster-ip-range: specifies the IP address segment of the service cluster.
- –service-node-port-range: specifies the port range of NodePort.
Note: If the kube-apiserver machine is not running kube-proxy, add — enable-Aggregator-routing =true.
Note: The CA certificate specified by requestheader-client-ca-file must have client auth and server Auth.
If — requesTheader-allowed-names is left blank or –proxy-client-cert-file is not in the CN name of the allowed-names certificate, Metrics of node or Pods fail to be checked.
1 [root@master01 ~]# kubectl top nodes
2 Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope
Copy the code5.2 distribute systemd
1 [root@master01 ~]# cd /opt/k8s/work 2 [root@master01 work]# source /root/environment.sh 3 [root@master01 work]# for (( i=0; i < 3; i++ )) 4 do 5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service 6 done 7 [root@master01 work]# ls 9 [root@master01 work]# for master_ip in ${MASTER_IPS[@]} 10 do 11 echo ">>> ${master_ip}" 12 scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service 13 Done # distribute systemdCopy the codeWarning: You need to perform this step only on master01.
Start and verify
6.1 Starting the Kube-Apiserver Service
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
7 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
8 done
Copy the codeWarning: You need to perform this step only on master01.
6.2 Checking kube-Apiserver Service
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "systemctl status kube-apiserver |grep 'Active:'"
7 done
Copy the codeWarning: You need to perform this step only on master01.
6.3 Checking Data Written to ETCD by Kube-Apiserver
1 [root@master01 ~]# cd /opt/k8s/work
2 [root@master01 work]# source /root/environment.sh
3 [root@master01 work]# ETCDCTL_API=3 etcdctl \
4 --endpoints=${ETCD_ENDPOINTS} \
5 --cacert=/opt/k8s/work/ca.pem \
6 --cert=/opt/k8s/work/etcd.pem \
7 --key=/opt/k8s/work/etcd-key.pem \
8 get /registry/ --prefix --keys-only
Copy the codeWarning: You need to perform this step only on master01.
6.4 Checking Cluster Information
1 [root@master01 work]# kubectl cluster-info 2 [root@master01 work]# kubectl get all --all-namespaces 3 [root@master01 4 [work] # kubectl get componentstatuses root @ master01 work] # netstat LNPT | grep 6443 # check kube - apiserver listening portsCopy the code
Tip:
If you execute kubectl imperative output when the following error message, then use the ~ /. Kube/config file is wrong, check the 009. Kubernetes binary master node deployment kubectl is properly completed, and then execute the command:
The connection to the server localhost:8080 was refused – did you specify the right host or port?
When kubectl get ComponentStatuses is executed, apiserver sends requests to 127.0.0.1 by default. At present, controller-Manager and Scheduler are not deployed, which is Unhealthy.
6443: Security port that receives HTTPS requests and authenticates and authorizes all requests.
16443: Nginx reverse proxy listening port;
8080 is not listened on because the insecure port is closed.
6.5 license
Grant kube-Apiserver access to the Kubelet API.
When executing kubectl exec, run, logs, etc., Apiserver forwards the request to Kubelet’s HTTPS port. This experiment defines RBAC rules to grant apiserver’s certificate (kubernetes.pem) user name (CN: kuberntes) access to kubelet API:
1 [root@master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
Copy the codeWarning: You need to perform this step only on master01.