Background:

Network environment reference:Cloud Networking ExperienceTwo VPC networks in Shanghai and Beijing. The server distribution is as follows:Explain why to use TencentOS Server 3.1 (TK4) system. Not because centos8 does not provide long-term maintenance…. , while experiencing Tencent Cloud open source Tencentos. See Tencent Cloud official website for details:Cloud.tencent.com/document/pr…. Comparison is compatible with Centos8 after all is compatible with CentoS8, according to centoS8 build Kubernetes process to build a Kubernetes experience across the region is feasible!

Basic planning:

Note: Well multi-area scatter comparison can also be highly available!

ip	hostname	area
10.10.2.8	sh-master-01	Shanghai 2 area
10.10.2.10	sh-master-02	Shanghai 2 area
10.10.5.4	sh-master-03	The five districts of Shanghai
10.10.4.7	sh-work-01	Shanghai 4 areas
10.10.4.14	sh-work-02	Shanghai 4 areas
10.10.12.9	bj-work-01	The five districts of Beijing

Create an Intranet SLB to be the VIP of Apiserver. The traditional SLB has been used in the past. Now only the application SLB is used……

System initialization

Note: Perform operations 1-12 for all nodes

1. Change the host name

Note: Host name is not initialized to modify the host name

[root@VM-2-8-centos ~]# hostnamectl set-hostname sh-master-01
[root@VM-2-8-centos ~]# cat /etc/hostname
sh-master-01
Copy the code

The others did the same

2. Disable the swap partition

swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
Copy the code

3. Close the selinux

[root@sh-master-01 ~]# setenforce  0 
ssive/SELINUX=disabled/g" /etc/selinux/configsetenforce: SELinux is disabled
[root@sh-master-01 ~]# sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
[root@sh-master-01 ~]# sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
[root@sh-master-01 ~]# sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
[root@sh-master-01 ~]# sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config
Copy the code

4. Disable the firewall

systemctl disable --now firewalld
chkconfig firewalld off
Copy the code

Note: Firewalld and iptables are not installed and can be ignored

5. Adjust the number of open files

cat> /etc/security/limits.conf <<EOF
* soft nproc 1000000
* hard nproc 1000000
* soft nofile 1000000
* hard nofile 1000000
* soft  memlock  unlimited
* hard memlock  unlimited
EOF
Copy the code

The tencentos limits.d directory contains 80-nofile.conf, where all configuration files can be stored. This avoids modifying the main file

6. yum update

yum update yum -y install gcc bc gcc-c++ ncurses ncurses-devel cmake elfutils-libelf-devel openssl-devel flex* bison* autoconf automake zlib* fiex* libxml* ncurses-devel libmcrypt* libtool-ltdl-devel* make cmake pcre pcre-devel openssl openssl-devel jemalloc-devel tlc libtool vim unzip wget lrzsz bash-comp* ipvsadm ipset jq sysstat conntrack libseccomp conntrack-tools socat curl wget git conntrack-tools psmisc nfs-utils tree bash-completion conntrack libseccomp net-tools  crontabs sysstat iftop nload strace bind-utils tcpdump htop telnet lsofCopy the code

Of course I omitted…… here My CVM initialization is usually done with oneInstack’s script

7. Ipvs added

The system kernel of tencentos is 5.4.119

:> /etc/modules-load.d/ipvs.conf module=( ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack br_netfilter ) for kernel_module in ${module[@]}; do /sbin/modinfo -F filename $kernel_module |& grep -qv ERROR && echo $kernel_module >> /etc/modules-load.d/ipvs.conf ||  : doneCopy the code

systemctl daemon-reload
systemctl enable --now systemd-modules-load.service
Copy the code

Verify that ipvS is successfully loaded

# lsmod | grep ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  5
ip_vs                 151552  11 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          114688  5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         20480  2 nf_conntrack,ip_vs
Copy the code

8. Optimize system parameters (not necessarily optimal, take all)

Oneinstack is installed initially by default. Cat /etc/sysctl.d/99-sysctl.conf

fs.file-max=1000000 net.ipv4.tcp_max_tw_buckets = 6000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_max_syn_backlog = 16384 net.core.netdev_max_backlog = 32768 net.core.somaxconn = 32768 net.core.wmem_default = 8388608 net.core.rmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_fin_timeout =  20 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.tcp_syncookies = 1 #net.ipv4.tcp_tw_len = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_mem = 94500000 915000000 927000000 net.ipv4.tcp_max_orphans = 3276800 net.ipv4.ip_local_port_range = 1024 65000 net.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_established = 3600Copy the code

9. Containerd installation

DNF/YUM Centos8 changes Pretty much……. Add ali cloud source custom as follows:

dnf install dnf-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo sudo yum update -y && sudo yum install -y containerd.io Containerd config default > / etc/containerd/config toml # replace containerd default sand_box mirror, Edit/etc/containerd/config. Toml sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2" restart the containerd $# systemctl daemon-reload $ systemctl restart containerdCopy the code

I can’t seem to get…. The matching version is not correct.Check out Tencent’s source and delete Ali’s first, of course:

rm -rf /etc/yum.repos.d/docker-ce.repo
yum clean all
Copy the code

Mirrors.cloud.tencent.com/docker-ce/l…

dnf install dnf-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.cloud.tencent.com/docker-ce/linux/centos/docker-ce.repo sudo yum update -y && sudo yum install -y Containerd. IO containerd config default > / etc/containerd/config toml # replace containerd default sand_box mirror, Edit/etc/containerd/config. Toml sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2" restart the containerd $# systemctl daemon-reload $ systemctl restart containerdCopy the code

Still so……. Without their own matching system…. How to do? Manually modify it?Tencentos can also support common yum sources. Don’t make me do it manually

containerd config default > /etc/containerd/config.toml
Copy the code

-reload systemctl restart containerd systemctl status containerdCopy the code

10. Configure the CRI client crictl

Note: there seems to be a version match

VERSION = "v1.22.0" wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz sudo tar zxvf  crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin rm -f crictl-$VERSION-linux-amd64.tar.gzCopy the code

Also may not move, github download to the desktop, manually upload it….

cat <<EOF > /etc/crictl.yaml runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: Crictl pull nginx:alpine crictl rmi nginx:alpine crictl imagesCopy the code

well Modify the/etc/containerd/config. Toml in * * [plugins. “IO. Containerd. GRPC. V1. Cri”. The registry. The mirrors. “docker. IO”]In theEndpoint ** is the address of ali Cloud accelerator (of course, it can also be other accelerators), in addition,[plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]Also add theSystemdCgroup = true Endpoint changed to Ali Cloud accelerator address:2lefsjdg.mirror.aliyuncs.com Restart the containerd service and download the image again.

systemctl restart containerd.service
crictl  pull nginx:alpine
Copy the code

OK

11. Install Kubeadm(centos8 does not have centos7 yum source)

Note: Why install version 1.21.3? Because I also have version 1.21.3 online. Just in time to test the upgrade

cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF # to delete the old version, Yum remove kubeadm kubectl kubelet kubernetes-cni zi-tools socat # yum remove kubeadm kubectl kubelet kubernetes-cni zi-tools socat # yum remove kubeadm kubectl kubelet kubernetes-cni zi-tools socat # yum list --showduplicates Kubeadm --disableexcludes=kubernetes # Install the specified version using the following command # yum -y install kubeadm-1.21.3 kubectl-1.21.3 kubelet-1.21.3 or # Install the latest stable version by default, 1.22.4 #yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes # start from startup systemctl enable kubelet.serviceCopy the code

Of course, here can also directly use the Tencent cloud source…. Same idea.

12. Modify kubelet configuration

vi /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS= --cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock
Copy the code

Master node additional operations:

1. Install haproxy

Note: HaProxy must be installed on the three master nodes and related configurations……

yum install haproxy
Copy the code

cat <<EOF > /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in  # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the Following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket  stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode tcp log global option tcplog option Dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend kubernetes Bind *:8443 # Configure port 8443 mode TCP default_backend kubernetes #--------------------------------------------------------------------- # static backend for serving up images, Stylesheets and to # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- backend kubernetes # backend server, This means that a call to 10.3.2.12:6443 will forward the request to the three servers on the back end, Balance Roundrobin Server Master1 10.10.2.8:6443 check MaxConn 2000 Server Master2 10.10.2.10:6443 check Maxconn 2000 Server master3 10.10.5.4:6443 check MaxConn 2000 EOF SystemCTL enable haProxy && Systemctl start haProxy &&  systemctl status haproxyCopy the code

Log in Tencent Cloud load balancing management background:console.cloud.tencent.com/clb, create a TCP listener named K8S to listen on port 6443. The back-end service is bound to three master nodes on port 8443. The default weight is 10.

2. The sh-master-01 node generates a configuration file

Note: The sh-master-02 or sh-master-03 node can also be used

kubeadm config print init-defaults > config.yaml
Copy the code

Modify the configuration file as follows:

apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: AdvertiseAddress: 10.10.2.8 bindPort: 6443 nodeRegistration: criSocket: / run/containerd/containerd sock name: sh-master-01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s certSANs: - sh-master-01 - sh-master-02 - sh-master-03 - sh-master.k8s. IO - localhost - 127.0.0.1-10.10.2.8-10.10.2.10 - 10.10.5.4-10.10.2.4-xx.xx.xx. xx apiVersion: kubeadm.k8s. IO/v1Beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "10.10.2.4:6443" controllerManager: {} DNS: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: 1.21.3 Networking: dnsDomain: cluster.local serviceSubnet: 172.31.0.0/16 scheduler: {} - apiVersion: kubeproxy. Config. K8s. IO/v1alpha1 kind: KubeProxyConfiguration mode: ipvs ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "rr" strictARP: false syncPeriod: 15s iptables: masqueradeAll: true masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30sCopy the code

Added the configuration of IPVS, specified the subnet of service, as well as the domestic mirror warehouse, XX.xx.xx. xx, I reserved an IP (IP can be reserved for future expansion of the master node at least).

3. Initialize the kubeadm master-01 node

kubeadm init --config /root/config.yaml
Copy the code

Note: The screenshot below doesn’t match the command above, because I started to want to install Cilium to… And it failed hahaha let’s try Calico firstYeah, not when I was optimizing the system parametersnet.ipv4.ip_forwardFor the record, sysctl-w is temporary

sysctl -w net.ipv4.ip_forward=1
Copy the code

Add the following to the configuration file:

cat <<EOF > /etc/sysctl.d/99-sysctl.conf
fs.file-max=1000000
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_max_syn_backlog = 16384
net.core.netdev_max_backlog = 32768
net.core.somaxconn = 32768
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_fin_timeout = 20
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_syncookies = 1
#net.ipv4.tcp_tw_len = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.ip_local_port_range = 1024 65000
net.nf_conntrack_max = 6553500
net.netfilter.nf_conntrack_max = 6553500
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
EOF

sysctl --system
Copy the code

Note: Execute for all nodes

kubeadm init --config /root/config.yaml 
Copy the code

4. Sh-master-02,sh-master-03 Controller plane nodes are added to the cluster

mkdir -p $HOME/.kube mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/. Kube /config Add sh-master-03 to the cluster. Add ca.* sa.* front-proxy-ca.* etcd/ca* in the sh-master-01 /etc/kubernetes/pki directory Kubeadm join 10.10.2.4:6443 --token abcdef.0123456789abcdef was distributed to sh-master-02 and sh-master-03 /etc/kubernetes/pki --discovery-token-ca-cert-hash sha256:ccfd4e2b85a6a07fde8580422769c9e14113e8f05e95272e51cca2f13b0eb8c3 --control-plan Then run the following command as sh-master-01:  mkdir -p $HOME/.kube sudo \cp /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configCopy the code

kubectl get nodes
Copy the code

Well, since the CNI network plug-in is not installed, the state is Notready.

The Work node is added to the cluster

Kubeadm join 10.10.2.4:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:ccfd4e2b85a6a07fde8580422769c9e14113e8f05e95272e51cca2f13b0eb8c3Copy the code

​

First, CNN management Console bought 1Mbps bandwidth first, after all, to do a test:

Install the CNI network plug-in

Flannel Cilium: Flannel Cilium, flannel Cilium, Flannel Cilium Run first and count one by one. Other later slowly learn to optimize)

The curl https://docs.projectcalico.org/v3.11/manifests/calico.yaml - OCopy the code

sed -i -e "s? 192.168.0.0/16? 172.31.0.0/16? g" calico.yamlCopy the code

kubectl apply -f calico.yaml 
kubectl get pods -o kube-system -o wide
Copy the code

Note: I am also extra in Tencent cloudPrivate network consoleAuxiliary CIDR was added. I wonder if I can communicate with container networks in other regions? Not tested yet…. Just want to add: [

] (Console.cloud.tencent.com/vpc/vpc?rid…)

Here’s a simple ping test:

1. Deploy two PODS in Shanghai

cat<<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx:alpine name: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: Containers: - name: busybox image: busybox:1.28.4 Command: -sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always EOFCopy the code

Yeah, they’re all in Shanghai

[root@sh-master-01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES Busybox 1/1 Running 14 14h 172.31.45.132 sh-work-01 <none> <none> nginx-7fb7FD49b4-zrg77 1/1 Running 0 14h 172.31.45.131  sh-work-01 <none> <none>Copy the code

2. NodeSelector scheduling starts a POD in Beijing District

Then I also want to start a POD to run in Beijing, what? Steal a lazy tag, nodeSelector dispatch it!

kubectl label node bj-work-01  zone=beijing
Copy the code

cat nginx1.yaml

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx1 name: nginx1 spec: nodeSelector: Zone: "Beijing" containers: - image: nginx name: nginx1 resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {}Copy the code

kubectl apply -f nginx1.yaml
Copy the code

[root@sh-master-01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES Busybox 1/1 Running 14 14h 172.31.45.132 sh-work-01 <none> <none> nginx-7fb7FD49b4-zrg77 1/1 Running 0 14h 172.31.45.131 Sh -work-01 <none> <none> nginx1 1/1 Running 0 14h 172.31.89.194 bj-work-01 <none> <none>Copy the code

3. The ping test

Ping pod from Beijing and Pod from Shanghai on sh-master-02Exec pod ping of Shanghai and BeijingIt’s pretty much the same. The main purpose is to verify whether it is feasible to build kubernetes cluster across regional VPC. We haven’t figured out how to test network quality yet. It’s just throwing stones. The clouds are much easier on the top. At least BGP configuration is relatively omitted. If there is a cloud across the region to build kubernetes cluster can refer to it.