The content is from the official Longhorn 1.1.2 English technical manual.

A series of

  • What’s a Longhorn?
  • Longhorn Cloud Native distributed block storage solution design architecture and concepts

The installation

Longhorn can be installed on a Kubernetes cluster in a number of ways:

  • Rancher catalog app
  • kubectl
  • Helm

Installation requirements

Each node in the Kubernetes cluster where Longhorn is installed must meet the following requirements:

  • withKubernetesCompatible container runtime (The Docker v1.13 +,Containerd v1.3.7 +Etc.)
  • Kubernetes v1.16 +.
    • Recommend Kubernetes v1.17 +
  • open-iscsiHas been installed andiscsidThe daemon is running on all nodes. It’s necessary becauseLonghornDepends on the hostiscsiadm 为 KubernetesPersistent volumes are provided.
  • RWX supportEach node is required to be installedNFSv4 client.
  • Host file system supportfile extentsFunction to store data. Currently we support:
    • ext4
    • XFS
  • curl.findmnt.grep.awk.blkid.lsblkMust be installed.
  • Mount Propagation must be enabled.

Longhorn Workloads must be able to run as root to properly deploy and operate Longhorn.

Operating system (OS)/ Release (Distro) Specific configuration

  • Google Kubernetes Engine (GKE) LonghornSome extra Settings are required to work properly.
  • The K3s clusters require some additional Settings.
  • RKE clusters with CoreOSNeed to becsi-on-rke-and-coreos

Use the Environment Check Script

We wrote a script to help you gather enough information about these factors.

Note that you may need to install JQ locally before running the Env Check script.

Run the script:

curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v{{< current-version >}}/scripts/environment_check.sh | bash
Copy the code

Example result:

daemonset.apps/longhorn-environment-check created
waiting for pods to become ready (0/3)
all pods ready (3/3)

  MountPropagation is enabled!

cleaning up...
daemonset.apps "longhorn-environment-check" deleted
clean up complete
Copy the code

Pod Security Policy

As of V1.0.2, Longhorn ships with the default Pod security policy, which will give Longhorn the necessary permissions to run properly.

Longhorn requires no special configuration to work on clusters with Pod security policies enabled.

Pay attention to the Mount Propagation

If your Kubernetes cluster is provided by Rancher V2.0.7 + or later, MountPropagation is enabled by default.

If MountPropagation is disabled, the Base Image function is disabled.

Install the open – iscsi

The commands used to install open-iscsi vary with Linux distributions.

For GKE, we recommend using Ubuntu as the guest OS image because it already includes Open-iscsi.

You may need to edit the Cluster Security Group to allow SSH access.

For SUSE and openSUSE, use the following command:

zypper install open-iscsi
Copy the code

For Debian and Ubuntu, use the following command:

apt-get install open-iscsi
Copy the code

For RHEL, CentOS, and EKS with Kubernetes Worker AMI with AmazonLinux2 Image, use the following command:

yum install iscsi-initiator-utils
Copy the code

We also provide an iscsi installer that makes it easier for users to automatically install Open-iscsi:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v{{< current-version >}}/deploy/prerequisite/longhorn-iscsi-installation.yaml
Copy the code

After deployment is complete, run the following command to check the pod status of the installer:

kubectl get pod | grep longhorn-iscsi-installation
longhorn-iscsi-installation-49hd7   1/1     Running   0          21m
longhorn-iscsi-installation-pzb7r   1/1     Running   0          39m
Copy the code

You can also run the following command to view the installation result:

kubectl logs longhorn-iscsi-installation-pzb7r -c iscsi-installation ... Installed: iscsi-initiator-utils.x86_64 0:6.2.0.874-7.amzn2 Dependency Installed: iscsi-initiator-utils.x86_64 0:6.2.0.874-7.amzn2 Dependency Installed: Iscsi initiator - utils - iscsiuio. X86_64 0-6. 2.0.874-7. Amzn2 Complete! Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service. iscsi install successfullyCopy the code

Install something that the client

The commands used to install NFSv4 Client vary from Linux distribution to Linux distribution.

For Debian and Ubuntu, use the following command:

apt-get install nfs-common
Copy the code

For RHEL, CentOS, and EKS with Kubernetes Worker AMI with AmazonLinux2 Image, use the following command:

yum install nfs-utils
Copy the code

We also provide an NFS installer that makes it easier for users to automatically install NFS-Client:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v{{< current-version >}}/deploy/prerequisite/longhorn-nfs-installation.yaml
Copy the code

After deployment is complete, run the following command to check the pod status of the installer:

kubectl get pod | grep longhorn-nfs-installation
NAME                                  READY   STATUS    RESTARTS   AGE
longhorn-nfs-installation-t2v9v   1/1     Running   0          143m
longhorn-nfs-installation-7nphm   1/1     Running   0          143m
Copy the code

You can also run the following command to view the installation result:

kubectl logs longhorn-nfs-installation-t2v9v -c nfs-installation
...
nfs install successfully
Copy the code

Check Kubernetes version

Use the following command to check your Kubernetes server version

kubectl version
Copy the code

Results:

Client Version: Version. The Info {Major: "1", Minor: "19", GitVersion: "v1.19.3 GitCommit:" 1 e11e4a2108024935ecfcb2912226cedeafd99df." BuildDate GitTreeState: "clean" : "the 2020-10-14 T12: graze Z," GoVersion: "go1.15.2", the Compiler: "gc", Platform:"linux/amd64"} Server Version: Version. The Info {Major: "1", Minor: "17", GitVersion: "v1.17.4 GitCommit:" 8 d8aa39598534325ad77120c120a22b3a990b5ea ", GitTreeState:"clean", BuildDate:" 2020-03-12T20:55:23z ", GoVersion:" GO1.13.8 ", Compiler:" GC ", Platform:" Linux/AMD64 "}Copy the code

The Server Version should be V1.16 or later.

Install as the Rancher Catalog App

One of the benefits of installing Longhorn through the Rancher Catalog is that Rancher provides authentication for the Longhorn UI.

If a new version of Longhorn is Available, you will see the Upgrade Available flag on the Catalog Apps screen. You can Upgrade Longhorn Manager by clicking the Upgrade button.

The installation

  1. Optional: We recommend thatLonghornCreate a new project, for exampleStorage.
  2. Navigate to the installation you will be installingLonghorn 的 cluster 和 project.

3. To navigate toCatalog AppsThe screen.In 4.catalogFound in theLonghornProject and click on it.5. Optional: Customize the default Settings. 6. Click theLaunch. LonghornWill be installed inlonghorn-systemNamespace.

Longhorn is now installed.

7. Click theindex.htmlLink navigation toLonghornThe dashboard.

After Longhorn is successfully installed, you can access the Longhorn UI by navigating to the Catalog Apps screen.

Install using Kubectl

Install the Longhorn

  1. Install Longhorn on any Kubernetes cluster using the following command:

    kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v{{< current-version >}}/deploy/longhorn.yaml
Copy the code

    One way to monitor installation progress is to observe the pod created in the Longhorn-system namespace:

    kubectl get pods \
--namespace longhorn-system \
--watch
Copy the code

  2. Check whether the deployment is successful:

    $ kubectl -n longhorn-system get podNAME READY STATUS RESTARTS AGE csi-attacher-6fdc77c485-8wlpg 1/1 Running 0 9d csi-attacher-6fdc77c485-psqlr 1/1 Running 0 9d csi-attacher-6fdc77c485-wkn69 1/1 Running 0 9d csi-provisioner-78f7db7d6d-rj9pr 1/1 Running 0 9d csi-provisioner-78f7db7d6d-sgm6w 1/1 Running 0 9d csi-provisioner-78f7db7d6d-vnjww 1/1 Running 0 9d engine-image-ei-6e2b0e32-2p9nk 1/1 Running 0 9d engine-image-ei-6e2b0e32-s8ggt 1/1 Running 0 9d engine-image-ei-6e2b0e32-wgkj5 1/1 Running 0 9d longhorn-csi-plugin-g8r4b 2/2 Running 0 9d longhorn-csi-plugin-kbxrl 2/2  Running 0 9d longhorn-csi-plugin-wv6sb 2/2 Running 0 9d longhorn-driver-deployer-788984b49c-zzk7b 1/1 Running 0 9d longhorn-manager-nr5rs 1/1 Running 0 9d longhorn-manager-rd4k5 1/1 Running 0 9d longhorn-manager-snb9t 1/1 Running 0 9d longhorn-ui-67b9b6887f-n7x9q 1/1 Running 0 9dCopy the code

  3. To enable access to the Longhorn UI, you need to set up an Ingress Controller. Authentication for Longhorn UI is not enabled by default.

List of deployed resources

The following projects will be deployed to Kubernetes:

Namespace: longhorn-system

All Longhorn bits will apply to this namespace.

ServiceAccount: longhorn-service-account

The Service Account is created in the Longhorn-system namespace.

ClusterRole: longhorn-role

This role will have access to:

  • In apiextension.k8s.io (All verbs)
    • customresourcedefinitions
  • In core (All verbs)
    • pods
      • /logs
    • events
    • persistentVolumes
    • persistentVolumeClaims
      • /status
    • nodes
    • proxy/nodes
    • secrets
    • services
    • endpoints
    • configMaps
  • In core
    • namespaces (get, list)
  • In apps (All Verbs)
    • daemonsets
    • statefulSets
    • deployments
  • In batch (All Verbs)
    • jobs
    • cronjobs
  • In storage.k8s.io (All verbs)
    • storageclasses
    • volumeattachments
    • csinodes
    • csidrivers
  • In coordination.k8s.io
    • leases
ClusterRoleBinding: longhorn-bind

This connects the Longhorn-Role to the Longhorn-service-Account in the Longhorn-System namespace.

CustomResourceDefinitions

To install the following CustomResourceDefinitions

  • In longhorn.io
    • engines
    • replicas
    • settings
    • volumes
    • engineimages
    • nodes
    • instancemanagers
Kubernetes API objects
  • One with default Settingsconfig map
  • longhorn-manager DaemonSet
  • longhorn-backendService will be internallonghorn-manager DaemonSetExposed to theKubernetes
  • longhorn-ui Deployment
  • longhorn-frontendService will be internallonghorn-uiExposed to theKubernetes
  • longhorn-driver-deployerThe deployment of CSI driver
  • longhorn StorageClass

Installation using Helm

Precautions for installing the Helm

For help installing Helm, see the official documentation.

If you are using a pre-3.0 version of Helm, you need to install Tiller in a Kubernetes cluster using role-based Access Control (RBAC).

Install the Longhorn

  1. Add Longhorn Helm repository:

    helm repo add longhorn https://charts.longhorn.io
Copy the code

  2. Get the latest Charts from the repository:

    helm repo update
Copy the code

  3. Install Longhorn in the Longhorn-system namespace. To install Longhorn using Helm 2, use the following command:

    helm install longhorn/longhorn --name longhorn --namespace longhorn-system
Copy the code

    To install Longhorn using Helm 3, use the following command:

    kubectl create namespace longhorn-system
helm install longhorn longhorn/longhorn --namespace longhorn-system
Copy the code

  4. To confirm the successful deployment, run:

    kubectl -n longhorn-system get pod
Copy the code

    The result should look like this:

    NAME READY STATUS RESTARTS AGE compatible-csi-attacher-d9fb48bcf-2rzmb 1/1 Running 0 8m58s csi-attacher-78bf9b9898-grn2c  1/1 Running 0 32s csi-attacher-78bf9b9898-lfzvq 1/1 Running 0 8m59s csi-attacher-78bf9b9898-r64sv 1/1 Running 0 33s csi-provisioner-8599d5bf97-c8r79 1/1 Running 0 33s csi-provisioner-8599d5bf97-fc5pz 1/1 Running 0 33s csi-provisioner-8599d5bf97-p9psl 1/1 Running 0 8m59s csi-resizer-586665f745-b7p6h 1/1 Running 0 8m59s csi-resizer-586665f745-kgdxs 1/1 Running 0 33s csi-resizer-586665f745-vsvvq 1/1 Running 0 33s engine-image-ei-e10d6bf5-pv2s6 1/1 Running 0 9m30s instance-manager-e-379373af 1/1 Running 0 8m41s instance-manager-r-101f13ba 1/1 Running 0 8m40s longhorn-csi-plugin-7v2dc 4/4 Running 0 8m59s longhorn-driver-deployer-775897bdf6-k4sfd 1/1 Running 0 10m longhorn-manager-79xgj 1/1 Running 0 9m50s longhorn-ui-9fbb5445-httqf 0/1 Running 0 33sCopy the code

  5. To enable access to the Longhorn UI, you need to set up an Ingress Controller. Authentication for Longhorn UI is not enabled by default.

Visit the UI

Prerequisites for access and authentication

These instructions assume that Longhorn has been installed.

If you have the Longhorn YAML manifest installed, the Ingress Controller needs to be set up to allow external traffic into the cluster, and authentication is not enabled by default. This applies to Helm and Kubectl installations.

If Longhorn is installed as the Rancher Catalog app, Rancher will automatically create an Ingress Controller for you with access control (Rancher-proxy).

Visit the Longhorn UI

After installing Longhorn in your Kubernetes cluster, you can access the UI Dashboard.

  1. Obtain Longhorn’s external service IP address:

    kubectl -n longhorn-system get svc
Copy the code

    For Longhorn V0.8.0, the output should look like this and the Longhorn UI should be accessed using the cluster-IP of Longhorn-Frontend:

    NAME TYPE cluster-ip external-ip PORT(S) AGE Longhorn-Backend ClusterIP 10.20.248.250 < None > 9500/TCP 58M Longhorn-frontend ClusterIP 10.20.245.110 < None > 80/TCP 58MCopy the code

    In the example above, the IP is 10.20.245.110.

    For Longhorn V0.8.0 +, the UI Service type is changed from LoadBalancer to ClusterIP.

  2. Navigate to the IP of the Longhorn-frontend in your browser.

    The Longhorn UI looks like this:

Create the Ingress using basic Authentication (Nginx)

If you install Longhorn on a Kubernetes cluster using Kubectl or Helm, you need to create an Ingress to allow external traffic to reach the Longhorn UI.

Kubectl and Helm installations are not authenticated by default. In these steps, you’ll learn how to create an Ingress with basic authentication using Annotations from the Nginx Ingress Controller.

  1. Create a basic authentication fileauth. The generated file is namedauthIs important (in fact)secretThere is a keydata.auth),Ingressreturn503. 
    $ USER=<USERNAME_HERE>; PASSWORD=<PASSWORD_HERE>; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth
Copy the code
  2. To create asecret: 
    $ kubectl -n longhorn-system create secret generic basic-auth --from-file=auth
Copy the code
  3. Create an Ingress listlonghorn-ingress.yml : 
    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
  annotations:
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # prevent the controller from redirecting (308) to HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80
Copy the code
  4. Create Ingress: 
    $ kubectl -n longhorn-system apply -f longhorn-ingress.yml
Copy the code

e.g.:

$ USER=foo; PASSWORD=bar; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth $ cat auth foo:$apr1$FnyKCYKb$6IP2C45fZxMcoLwkOwf7k0 $ kubectl -n longhorn-system create secret generic basic-auth --from-file=auth  secret/basic-auth created $ kubectl -n longhorn-system get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJEZueUtDWUtiJDZJUDJDNDVmWnhNY29Md2tPd2Y3azAK kind: Secret metadata: creationTimestamp: "2020-05-29T10:10:16Z" name: basic-auth namespace: longhorn-system resourceVersion: "2168509" selfLink: /api/v1/namespaces/longhorn-system/secrets/basic-auth uid: 9f66233f-b12f-4204-9c9d-5bcaca794bb7 type: Opaque $ echo " apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: longhorn-ingress namespace: longhorn-system annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # prevent the controller from redirecting (308) to HTTPS nginx.ingress.kubernetes.io/ssl-redirect: 'false' # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required ' spec: rules: - http: paths: - pathType: Prefix path: "/" backend: service: name: longhorn-frontend port: number: 80 " | kubectl -n longhorn-system create -f - ingress.networking.k8s.io/longhorn-ingress created $ kubectl -n Longhorn - system get ingress NAME HOSTS ADDRESS PORTS AGE longhorn - ingress * 45.79.165.114 66.228.45.37, 97.107.142.125 80 2m7s $curl -v http://97.107.142.125/ * Trying 97.107.142.125... TCP_NODELAY set * Connected to 97.107.142.125 (97.107.142.125) port 80 (#0) > GET/HTTP/1.1 > Host: 97.107.142.125 > user-agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: Openresty /1.15.8.1 < Date: Fri, 29 May 2020 11:47:33 GMT < Content-Type: text/ HTML < Content-Length: 185 < Connection: keep-alive < WWW-Authenticate: Basic realm="Authentication Required" < <html> <head><title>401 Authorization Required</title></head> <body> <center>< H1 >401 Authorization Required</ H1 ></center> < HR >< Center > OpenReSTY /1.15.8.1</ Center ></ body> </ HTML > * Connection #0 to host 97.107.142.125 left intact * Closing Connection 0 $curl -v http://97.107.142.125/ -u foo:bar * Trying 97.107.142.125... * TCP_NODELAY set * Connected to 97.107.142.125 (97.107.142.125) port 80 (#0) * Server auth using Basic with user 'foo' > GET/HTTP/1.1 > Host: 97.107.142.125 > Authorization: Basic Zm9vOmJhcg== > User-agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 29 May 2020 11:51:27 GMT < content-Type: text/ HTML < content-length: 1118 < Last-Modified: Thu, 28 May 2020 00:39:41 GMT < ETag: "5ecf084d-3fd" < Cache-Control: max-age=0 < <! DOCTYPE html> <html lang="en"> ......Copy the code

AWS EKS KubernetesAdditional steps for clustering

You will need to create an ELB (elastic load balancer) to expose the Nginx Ingress Controller to the Internet. Additional fees may be required.

  1. Create the necessary resources according to Nginx Ingress Controller Documentation.

  2. Follow the ingress-nginx/deploy/#aws step to create the ELB.

References

Kubernetes. Making. IO/ingress – ngi…

upgrade

Here, we show how to upgrade from all previous versions to the latest Longhorn.

Upgrade Longhorn

The upgrade process usually consists of two steps: first, upgrade the Longhorn Manager to the latest version, and then manually upgrade the Longhorn Engine to the latest version using the latest Longhorn Manager.

1. Upgrade Longhorn Manager

  • From you toV1.1. XUpgrade, seelonghorn-manager.

2. Manually upgrade the Longhorn Engine

After the Longhorn Manager upgrade, the Longhorn Engine also needs to be upgraded using the Longhorn UI.

3. Automatically upgrade the Longhorn Engine

Starting with Longhorn V1.1.1, we’ve provided an option to help you automatically upgrade the engine.

Note: There is an error in the instance manager image v1_20201216 provided in Longhorn V1.1.0 and V1.1.1 that can cause deadlock in a large cluster with hundreds of volumes. See more details at Longhorn /issues/2697. Longhorn V1.1.2 comes with a new instance manager image v1_20210621, which fixes deadlocks, but the volume’s engine/Replica process will not migrate from the old instance manager to the new instance manager, Until the next detached/attached volume. Longhorn did this because we didn’t want to interrupt the data plane of the volume.

If you are experiencing a deadlock in the old instance manager, follow the recovery steps for Issues /2697#issuecomment-879374809

Upgrade Longhorn Manager

fromV1.1. Xupgrade

We only support upgrading from V1.1.x to V1.1.2. For other versions, upgrade to V1.1. x first.

Support for real-time Engine upgrades from V1.1. x to V1.1.2.

For the AirGap upgrade when Longhorn is installed as the Rancher app, you need to change the image name and remove the Registry URL section.

Longhorn, for example, the mirror registry.example.com/longhorn/longhorn-manager:v1.1.2 change of section images of Longhorn/Longhorn – manager: v1.1.2.

Ready to upgrade

If Longhorn is installed using Helm Chart or as a Rancher Catalog app, check to ensure that the parameters in the default StorageClass have not changed. Changing the parameters of the default StorageClass may cause the Chart upgrade to fail. If you want to reconfigure the parameters in StorageClass, you can copy the default StorageClass configuration to create another StorageClass.

The current default StorageClass has the following parameters:

    parameters:
      numberOfReplicas: <user specified replica count, 3 by default>
      staleReplicaTimeout: "30"
      fromBackup: ""
      baseImage: ""
Copy the code

upgrade

Prerequisite: Always back up the volume before the upgrade. If anything goes wrong, you can use the backup to restore volumes.

To use kubectl upgrade, run the following command:

Kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/deploy/longhorn.yamlCopy the code

To use the Helm upgrade, run the following command:

helm upgrade longhorn ./longhorn/chart
Copy the code

On a Kubernetes cluster with Rancher 2.1 or newer version management, the steps for upgrading the Catalog app Longhorn-System are similar to the installation steps.

Then wait for all the pods to run and the Longhorn UI to work. Such as:

$ kubectl -n longhorn-system get pod
NAME                                        READY   STATUS    RESTARTS   AGE
csi-attacher-78bf9b9898-mb7jt               1/1     Running   1          3m11s
csi-attacher-78bf9b9898-n2224               1/1     Running   1          3m11s
csi-attacher-78bf9b9898-rhv6m               1/1     Running   1          3m11s
csi-provisioner-8599d5bf97-dr5n4            1/1     Running   1          2m58s
csi-provisioner-8599d5bf97-drzn9            1/1     Running   1          2m58s
csi-provisioner-8599d5bf97-rz5fj            1/1     Running   1          2m58s
csi-resizer-586665f745-5bkcm                1/1     Running   0          2m49s
csi-resizer-586665f745-vgqx8                1/1     Running   0          2m49s
csi-resizer-586665f745-wdvdg                1/1     Running   0          2m49s
engine-image-ei-62c02f63-bjfkp              1/1     Running   0          14m
engine-image-ei-62c02f63-nk2jr              1/1     Running   0          14m
engine-image-ei-62c02f63-pjtgg              1/1     Running   0          14m
engine-image-ei-ac045a0d-9bbb8              1/1     Running   0          3m46s
engine-image-ei-ac045a0d-cqvv2              1/1     Running   0          3m46s
engine-image-ei-ac045a0d-wzmhv              1/1     Running   0          3m46s
instance-manager-e-4deb2a16                 1/1     Running   0          3m23s
instance-manager-e-5526b121                 1/1     Running   0          3m28s
instance-manager-e-eff765b6                 1/1     Running   0          2m59s
instance-manager-r-3b70b0db                 1/1     Running   0          3m27s
instance-manager-r-4f7d629a                 1/1     Running   0          3m22s
instance-manager-r-bbcf4f17                 1/1     Running   0          2m58s
longhorn-csi-plugin-bkgjj                   2/2     Running   0          2m39s
longhorn-csi-plugin-tjhhq                   2/2     Running   0          2m39s
longhorn-csi-plugin-zslp6                   2/2     Running   0          2m39s
longhorn-driver-deployer-75b6bf4d6d-d4hcv   1/1     Running   0          3m57s
longhorn-manager-4j77v                      1/1     Running   0          3m53s
longhorn-manager-cwm5z                      1/1     Running   0          3m50s
longhorn-manager-w7scb                      1/1     Running   0          3m50s
longhorn-ui-8fcd9fdd-qpknp                  1/1     Running   0          3m56s
Copy the code

After the upgrade

To avoid an existing volume crash and switch from a disused set of Guaranteed Engine cpus to the New Instance Manager CPU reservation mechanism, Longhorn will automatically set Engine Manager CPU Request and Replica Manager CPU Request from each node based on deprecated Settings during the upgrade. Then, the new global instance Manager CPU Settings Guaranteed Engine Manager CPU and Guaranteed Replica Manager CPU will not take effect. You may want to review the new mechanism and setup instructions to see if any adjustments need to be made.

troubleshooting

Error: "longhorn" is invalid: provisioner: Forbidden: updates to provisioner are forbidden.

  • This means that some changes have been made to the default storageClass, and you need to clean up the old ones before upgrading.

  • To clean up the deprecated StorageClass, run the following command:

    Kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/examples/storageclass.yamlCopy the code

Upgrade the Longhorn Engine manually

In this section, you’ll learn how to manually upgrade the Longhorn Engine from the Longhorn UI.

A prerequisite for

Make sure you back up the Longhorn Engine image before upgrading it.

Upgrade Longhorn Manager before upgrading Longhorn Engine.

Note: There is an error in the instance manager image v1_20201216 provided in Longhorn V1.1.0 and V1.1.1 that can cause deadlock in a large cluster with hundreds of volumes. See more details at Longhorn /issues/2697. Longhorn V1.1.2 comes with a new instance manager image v1_20210621, which fixes deadlocks, but the volume’s Engine/Replica process will not migrate from the old instance manager to the new instance manager, Until the next detached/attached volume. Longhorn did this because we didn’t want to interrupt the data plane of the volume.

To reduce the chances of deadlocks occurring while the Engine /replica process is still in the old instance manager, you should upgrade the volume’s engine in small batches, for example, 2 or 3 volumes at a time.

Offline upgrade

If the real-time upgrade cannot be performed or the volume is degraded, perform the following steps:

  1. In accordance with theDetach procedure related to workloadsCarry on.
  2. Use batch selection to select all volumes. Click the batch operation buttonUpgrade Engine, select the available ones in the listengineThe mirror. This is the default engine shipped with this version manager.
  3. Restore allworkloads. Anything not belonging toKubernetes workloadAll volumes must be fromLonghorn UIAdditional.

In real time to upgrade

Upgrading from V1.1. x to V1.1.2 supports real-time upgrades.

The iSCSI front-end does not support real-time upgrade.

Real-time upgrades should only be performed on healthy volumes.

  1. Select the volume to be upgraded.
  2. Click in the drop-down menuUpgrade Engine.
  3. Select what you want to upgrade toengineThe mirror.
    1. It is usually unique in the listengineMirror image, becauseUIRemoves the current mirror from the list.
  4. Click on theOK.

During a live upgrade, users are temporarily presented with a double number of replicas. After the upgrade is complete, users should see the same number of replicas as before and update the Engine Image field of the volume.

Note that Rancher or Kubernetes will still display a new version of the engine’s old replica and replicas after a live upgrade. This is expected. If the image of the new version is listed as a Volume mirror on the Volume Detail page, the upgrade is successful.

Cleaning up old images

After all images have been upgraded, select Settings/Engine Image from the Longhorn UI. You should now be able to remove non-default images.

Automatic upgrade of Longhorn Engine

Starting with Longhorn V1.1.1, we provide an option to help you automatically upgrade the Longhorn volume to the new default engine version after upgrading Longhorn Manager. This feature reduces the manual work that must be done to upgrade Longhorn. Some related concepts of this feature are shown below:

1. Limit the number of concurrent automatic engine upgrades per node

This is a setting that controls how Longhorn automatically upgrades the volume’s engine to the new default engine image after upgrading Longhorn Manager. The value of this setting specifies the maximum number of engines that each node is allowed to upgrade to the default engine image simultaneously. If the value is 0, Longhorn will not automatically upgrade the volume engine to the default version. The higher this value is, the faster the engine upgrade process is completed.

However, providing a larger value for this setting can consume CPU and memory on more nodes during an engine upgrade. We recommend setting this value to 3 to leave some room for errors, but not to overwhelm the system with too many failed upgrades.

2. Longhorn’s behavior under different volume conditions.

In the following cases, the concurrent automatic engine upgrade per node limit is assumed to be greater than 0.

  1. Additional volume

    If the volume is attached and healthy, Longhorn will automatically upgrade the engine of the volume to the new default engine image in real time.

  2. Separation of volume

    Longhorn automatically upgrades detached volumes offline.

  3. Disaster volume

    Longhorn will not automatically upgrade Disaster Recovery Volumes to the new default engine image because it will trigger a full recovery of the disaster recovery volumes. A full recovery may affect the performance of other running Longhorn volumes on the system. Therefore, Longhorn lets you decide when it is a good time to manually upgrade the disaster recovery volume engine (for example, when the system is idle or during maintenance).

    However, when you activate the Dr Volume, it is activated and then separated. At this point, Longhorn will automatically upgrade the volume offline, similar to the case of detached volumes.

3. What happens if the upgrade fails?

If the volume upgrade engine fails, the engine image in the volume spec will remain different from the engine image in the volume state. Longhorn will retry the upgrade until it succeeds.

Longhorn will stop upgrading volumes on a node if there are too many volumes for each node to upgrade (i.e., exceeding the Concurrent automatic engine upgrade per node limit setting).

Uninstall the Longhorn

In this section, you will learn how to uninstall Longhorn.

  • A prerequisite for
  • Uninstall Longhorn from Rancher UI
  • Uninstall Longhorn using Helm
  • Uninstall Longhorn using Kubectl
  • troubleshooting

A prerequisite for

To prevent damage to the Kubernetes cluster, We recommend deleting all that use Longhorn volumes (PersistentVolume, PersistentVolumeClaim, StorageClass, Deployment, StatefulSet, DaemonSet, etc.) Kubernetes workload.

Uninstall Longhorn from Rancher UI

From the Rancher UI, navigate to the Catalog Apps TAB and delete the Longhorn App.

Uninstall Longhorn using Helm

Run this command:

helm uninstall longhorn -n longhorn-system
Copy the code

Uninstall Longhorn using Kubectl

  1. Create an uninstall job to remove CRDs from the system and wait for success:

    Kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/uninstall/uninstall.yaml kubectl get job/longhorn-uninstall -n default -wCopy the code

    Example output:

    $kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/uninstall/uninstall.yaml serviceaccount/longhorn-uninstall-service-account created clusterrole.rbac.authorization.k8s.io/longhorn-uninstall-role created clusterrolebinding.rbac.authorization.k8s.io/longhorn-uninstall-bind created job.batch/longhorn-uninstall created $ kubectl get job/longhorn-uninstall -n default -w NAME COMPLETIONS DURATION AGE longhorn-uninstall 0/1 3s 3s longhorn-uninstall 1/1 20s 20s ^CCopy the code

  2. Delete remaining components:

    Kubectl delete - f kubectl delete https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/deploy/longhorn.yaml - f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/uninstall/uninstall.yamlCopy the code

Tip: If you try kubectl delete – f {{< https://raw.githubusercontent.com/longhorn/longhorn/v current – version >}}/deploy/longhorn.yaml and stuck there, Please press Ctrl C and then run kubectl create – f {{< https://raw.githubusercontent.com/longhorn/longhorn/v current – version >}} / uninstall/uninstall yaml can also help you remove the Longhorn. Finally, don’t forget to clean up the remaining components.

troubleshooting

Instead of following the uninstall procedure, I removed the Longhorn application from the Rancher UI

Redeploy (same version) Longhorn App. Follow the uninstall procedure above.

The problem of CRDs

If your CRD instance or CRD itself cannot be deleted for some reason, run the following command to clean it up. Note: This clears all Longhorn states!

# Delete CRD finalizers, instances and definitions
for crd in $(kubectl get crd -o jsonpath={.items[*].metadata.name} | tr ' ' '\n' | grep longhorn.rancher.io); do
  kubectl -n ${NAMESPACE} get $crd -o yaml | sed "s/\- longhorn.rancher.io//g" | kubectl apply -f -
  kubectl -n ${NAMESPACE} delete $crd --all
  kubectl delete crd/$crd
done
Copy the code

Volumes can be attached/detached from the UI, but Kubernetes Pod/StatefulSet etc cannot use it

Check whether the volume plug-in directory is set correctly. Unless explicitly set by the user, it is automatically detected. Note: The FlexVolume plug-in has been deprecated since Longhorn V0.8.0 and should not be used again.

By default, Kubernetes use/usr/libexec/Kubernetes/kubelet – plugins/volume/exec, as stated in the official documentation.

Some vendors choose to change directories for a variety of reasons. GKE, for example, use the/home/kubernetes/flexvolume instead.

Users can run on the host ps aux | grep kubelet and check – volume – the plugin – dir parameters to find the correct directory. If not, will use the default/usr/libexec/kubernetes/kubelet – plugins/volume/exec.

See Longhorn/Longhorn for more information.