Author’s brief introduction

Hailong Wang, Rancher China Community Technology Manager, is responsible for the maintenance and operation of Rancher China Technology Community. I have 6 years of experience in cloud computing and experienced the technical transformation from OpenStack to Kubernetes. I have rich practical experience in operation and peacekeeping no matter the underlying operating system Linux, virtual KVM or Docker container technology.

Before the speech

K3s is a lightweight Kubernetes distribution. According to the statistics of K3s downloads in 2020, K3s has been downloaded more than 1 million times worldwide, with an average of more than 20,000 times installed per week, 30% of which are from China. Many users in China have applied K3s to various edge computing and Internet of Things devices, as well as production line robots deployed in smart factories and some of the world’s largest wind farms.

An insurmountable problem with K3s in production is offline installation. Prepare the following three components for your offline environment:

  • K3s installation script

  • Binary file of K3s

  • K3s depends on the mirror

All three components are available via the K3s Release page (github.com/k3s-io/k3s/… Mirror.cnrancher.com get these components.

I think the key of offline installation lies in the image part that K3s depends on, because the “installation script” and “binary file” of K3s only need to download to the corresponding directory, and then give the corresponding permission, very simple. However, how K3s relies on images to be installed depends on whether you’re using a manually deployed image or a private image repository, and whether the container is running using Containerd or Docker.

For different combinations, the offline installation can be implemented in the following ways:

  • Containerd + Manually deploy an image

  • Docker + Manually deploy images

  • Containerd + Private image repository mode

  • Docker + private image repository mode

Containerd + Manually deploy an image

Assume that you have downloaded the installation script (k3s-install.sh), binary file (K3s), and dependent image (k3s-airgap-images-amd64.tar) of the same version of K3s to /root.

If you use the runtime container for containerd, when start K3s, it checks the/var/lib/rancher/K3s/agent/images/whether there is any available image compression package, if present, will mirror the import to containerd image list. So we only need to download K3s dependent mirrored to/var/lib/rancher/K3s/agent/images/directory, then start K3s can.

1. Import the image to containerd Image list

sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp /root/k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
Copy the code

2. Move the K3s installation script and the K3s binary file to the corresponding directory and grant the executable permission

sudo chmod a+x /root/k3s /root/k3s-install.sh
sudo cp /root/k3s /usr/local/bin/
Copy the code

3. Install K3s

INSTALL_K3S_SKIP_DOWNLOAD=true /root/k3s-install.sh
Copy the code

Wait a moment, you can check that K3s has started successfully:

Root @ k3s - docker: ~ # crictl images the IMAGE TAG IMAGE ID SIZE docker. IO/rancher/coredns - coredns 1.8.0 comes with 296 a6d5035e2d 42.6 MB Docker. IO/rancher/klipper - helm v0.3.2 4 be09ab862d40 146 MB docker. IO/rancher/klipper - 897 lb v0.1.2 ce3c5fc8ff 6.46 MB Docker. IO/rancher/library - busybox 1.31.1 1 c35c44120825 1.44 MB docker. IO/rancher/library - traefik 1.7.19 aa764f7db3051 86.6 MB docker. IO/rancher/local path - provisioner v0.0.14 e422121c9c5f9 42 MB docker. IO/rancher/metrics - server v0.3.6 IO /rancher/pause 3.1 da86e6ba6CA19 746kB root@k3s-docker:~# kubectl get Pods -a NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-7c458769fb-zdg9z 1/1 Running 0 38s kube-system coredns-854c77959c-696gk 1/1 Running 0 38s kube-system metrics-server-86cbb8457f-hs6vw 1/1 Running 0 38s kube-system helm-install-traefik-4pgcr 0/1 Completed 0 38s kube-system svclb-traefik-bq7wl 2/2 Running 0 17s kube-system traefik-6f9cbd9bd4-jccd7 1/1 Running 0 17sCopy the code

Docker + Manually deploy images

Assume that you have downloaded the installation script (k3s-install.sh), binary file (K3s), and dependent image (k3s-airgap-images-amd64.tar) of the same version of K3s to /root.

Unlike containerd, use a docker as container is running, start the K3s not import/var/lib/rancher/K3s/agent/images/directory of images. Therefore, before starting K3s, we need to manually import the images that K3s depends on into the Docker image list.

1. Import the image to the Docker image list

sudo docker load -i /root/k3s-airgap-images-amd64.tar
Copy the code

2. Move the K3s installation script and the K3s binary file to the corresponding directory and grant the executable permission

sudo chmod a+x /root/k3s /root/k3s-install.sh
sudo cp /root/k3s /usr/local/bin/
Copy the code

3. Install K3s

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='--docker' /root/k3s-install.sh
Copy the code

Wait a moment, you can check that K3s has started successfully:

root@k3s-docker:~# Docker images REPOSITORY TAG IMAGE ID CREATED SIZE Rancher/Klipper - Helm V0.3.24be09AB862d4 7 weeks Ago 145MB Rancher/coreDNS - CoreDNS 1.8.0 296a6d5035e2 2 months ago 145MB Rancher/Library-busybox 1.31.1 1C35C4412082 7 Months ago 145MB Rancher/CoreDNS - CoreDNS 1.8.0 296a6d5035e2 2 Months ago 145MB Rancher/Library-busybox 1.31.1 1C35c4412082 7 Months ago 1.22MB Rancher /local-path-provisioner V0.0.14 e422121C9C5F 7 months ago 41.7MB Rancher/Library-Traefik 1.7.19 Aa764f7db305 14 months ago 85.7MB Rancher /metrics-server V0.3.6 9DD718864ce6 14 months ago 39.9MB Rancher /klipper-lb V0.1.2 897ce3C5FC8f 19 months ago 6.1MB rancher/pause 3.1 da86e6ba6ca1 3 years ago 742kB root@k3s-docker:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system metrics-server-86cbb8457f-8ckr6 1/1 Running 0 30s kube-system local-path-provisioner-7c458769fb-vhkjr 1/1 Running 0 30s kube-system helm-install-traefik-4b46c 0/1 Completed 0 31s kube-system coredns-854c77959c-4ql8t 1/1 Running 0 30s kube-system svclb-traefik-kbtbx 2/2 Running 0 27s  kube-system traefik-6f9cbd9bd4-rbm6k 1/1 Running 0 27sCopy the code

Containerd + Private image repository mode

Assume that you have downloaded the K3s installation script (k3s-install.sh) and the binary file (K3s) of the same version to /root. And K3s need mirror image has been uploaded to the warehouse (in this case a mirror image of the warehouse address is: http://192.168.64.44:5000). A list of images required by K3s can be obtained from k3s-images.txt on the K3s Release page.

1. Configure the K3s mirror vault

Starting K3s pulls images from docker.io by default. IO to install the container offline, configure the address of the Containerd image repository to the endpoint of the Docker. IO file. For details, see Configuring containerd Image Repository.

Docs. The rancher. Cn/docs/k3s/in…

sudo mkdir -p /etc/rancher/k3s
sudo cat >> /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "docker.io":
    endpoint:
      - "http://192.168.64.44:5000"
      - "https://registry-1.docker.io"
EOF
Copy the code

2. Move the K3s installation script and the K3s binary file to the corresponding directory and grant the executable permission

sudo chmod a+x /root/k3s /root/k3s-install.sh
sudo cp /root/k3s /usr/local/bin/
Copy the code

3. Install K3s

INSTALL_K3S_SKIP_DOWNLOAD=true /root/k3s-install.sh
Copy the code

Wait a moment, you can check that K3s has started successfully:

Root @ k3s - containerd: ~ # crictl images the IMAGE TAG IMAGE ID SIZE docker. IO/rancher/coredns - coredns a6d5035e2d 1.8.0 comes with 296 12.9 MB docker. IO/rancher/klipper - helm v0.3.2 4 be09ab862d40 50.7 MB docker. IO/rancher/klipper - 897 lb v0.1.2 ce3c5fc8ff 2.71 MB docker. IO/rancher/library - traefik 1.7.19 aa764f7db3051 24 MB docker. IO/rancher/local path - provisioner v0.0.14 E422121c9c5f9 13.4 MB docker. IO/rancher/metrics - server v0.3.6 9 dd718864ce61 10.5 MB docker. IO/rancher/pause 3.1 da86e6ba6ca19 326kB root@k3s-containerd:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system local-path-provisioner-7c458769fb-7w8hb 1/1 Running 0 37s kube-system coredns-854c77959c-f8m2n 1/1 Running 0 37s kube-system helm-install-traefik-9lbrx 0/1 Completed 0 38s kube-system svclb-traefik-x8f6f 2/2 Running 0 29s kube-system  metrics-server-86cbb8457f-f7lb7 1/1 Running 0 37s kube-system traefik-6f9cbd9bd4-4s66r 1/1 Running 0 29sCopy the code

Docker + private image repository mode

Assume that you have downloaded the K3s installation script (k3s-install.sh) and the binary file (K3s) of the same version to /root. And K3s need mirror image has been uploaded to the warehouse (in this case a mirror image of the warehouse address is: http://192.168.64.44:5000). A list of images required by K3s can be obtained from k3s-images.txt on the K3s Release page.

1. Configure the K3s mirror vault

Docker does not allow containerd to change the address of the default image repository indirectly by changing the endpoint corresponding to docker. IO (default: Registry-1.docker. IO). In Docker, however, you can configure Registry -mirrors to obtain K3s images from other mirrors. After this configuration, the mirrors will be pulled from the registry-mirrors configuration address first. If the mirrors cannot be pulled from the default docker. IO, the mirrors will be pulled from the default docker.

cat >> /etc/docker/daemon.json <<EOF { "registry-mirrors": [" http://192.168.64.44:5000 "]} EOF sudo systemctl daemon - reload sudo systemctl restart dockerCopy the code

2. Move the K3s installation script and the K3s binary file to the corresponding directory and grant the executable permission

sudo chmod a+x /root/k3s /root/k3s-install.sh
sudo cp /root/k3s /usr/local/bin/
Copy the code

3. Install K3s

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='--docker' /root/k3s-install.sh
Copy the code

Wait a moment, you can check that K3s has started successfully:

root@k3s-docker:~# Docker images REPOSITORY TAG IMAGE ID CREATED SIZE Rancher/Klipper - Helm V0.3.24be09AB862d4 7 weeks Ago 145MB Rancher/CoreDNS-CoreDNS 1.8.0 296A6D5035e2 2 months ago 145MB Rancher/Local-path-provisioner V0.0.14 E422121c9c5f 7 months ago 41.7MB Rancher/Library-traefik 1.7.19 aa764f7db305 14 months ago 85.7MB Rancher /metrics-server V0.3.6 9DD718864CE6 14 months ago 39.9MB Rancher /klipper- LB v0.1.2 897ce3C5FC8f 19 months ago 6.1MB Rancher /pause 3.1 da86e6ba6ca1 3 years ago 742kB root@k3s-docker:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system helm-install-traefik-bcclh 0/1 Completed 0 33s kube-system coredns-854c77959c-kp85f 1/1 Running 0 33s kube-system metrics-server-86cbb8457f-85fpd 1/1 Running 0 33s kube-system local-path-provisioner-7c458769fb-r5nkw 1/1 Running 0 33s kube-system svclb-traefik-rbmhk 2/2 Running 0 24s kube-system traefik-6f9cbd9bd4-k6t9n 1/1 Running 0 24sCopy the code

Remember after

Manual image deployment is suitable for small-scale installation with a small number of nodes. A private mirror warehouse is suitable for a large cluster with many nodes. Docker run -d -p 55:5000 –restart=always –name registry:2, You may need to modify some of the parameters of Registry in your environment because of the way the mirror repository is set up.

The resources

K3s offline installation documentation:

Docs. The rancher. Cn/docs/k3s/in…