[Kubernetes (K8S)] Helm install Rancher Labs Longhorn Cloud Native distributed block storage for Kubernetes (K8S)

Posted on Edited on In , , Views: Word count in article: 11k Reading time ≈ 10 mins.

Rancher Labs Longhorn

Longhorn is a lightweight, reliable and easy-to-use Cloud native distributed block storage system for Kubernetes.

Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation.

With Longhorn, you can:

  • Use Longhorn volumes as persistent storage for the distributed stateful applications in your Kubernetes cluster

  • Partition your block storage into Longhorn volumes so that you can use Kubernetes volumes with or without a cloud provider
    Replicate block storage across multiple nodes and data centers to increase availability

  • Store backup data in external storage such as NFS or AWS S3

  • Create cross-cluster disaster recovery volumes so that data from a primary Kubernetes cluster can be quickly recovered from backup in a second Kubernetes cluster

  • Schedule recurring snapshots of a volume, and schedule recurring backups to NFS or S3-compatible secondary storage
    Restore volumes from backup

  • Upgrade Longhorn without disrupting persistent volumes

  • Manipulate Longhorn resources with kubectl

This article is about how to use Helm to deploy Longhorn on Kubernetes (K8S).

Prerequisites

  • Kubernetes (K8S)
    Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

    For more information about installing and using Kubernetes (K8s), see the Kubernetes (K8s) Docs.

  • Helm
    Helm is the best way to find, share, and use software built for Kubernetes.

    For more information about installing and using Helm, see the Helm Docs.

  • open-iscsi is installed, and the iscsid daemon is running on all the nodes.
    The Open-iSCSI project provides a high-performance, transport independent, implementation of RFC 3720 iSCSI for Linux.

    You will get an error message without open-iscsi.

    1
    level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host"

    For help installing open-iscsi:

    1
    2
    3
    4
    5
    # For Debian and Ubuntu, use this command:
    $ sudo apt-get install open-iscsi

    # For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:
    $ sudo yum install iscsi-initiator-utils
    1
    2
    3
    # start and enable iscsid.service
    $ sudo systemctl start iscsid && sudo systemctl enable iscsid
    $ sudo systemctl status iscsid

Installation

Install by Helm

Helm install longhorn into longhorn-system namespace.

1
2
3
4
5
6
7
8
9
10
11
# crate namespace:
$ kubectl create namespace longhorn-system

# Add the Helm repository:
$ helm repo add longhorn https://charts.longhorn.io

# Update your local Helm chart repository cache:
$ helm repo update

# To install Helm chart:
$ helm install longhorn longhorn/longhorn --namespace longhorn-system -f values.yaml

See Helm release about longhorn.

1
2
3
$ helm list --namespace longhorn-system
NAME    	NAMESPACE      	REVISION	UPDATED                               	STATUS  	CHART         	APP VERSION
longhorn	longhorn-system	1       	2020-10-03 14:17:30.890902 +0800 +0800	deployed	longhorn-1.0.2	v1.0.2

See pods about longhorn.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ kubectl get pods -n longhorn-system
NAME                                        READY   STATUS              RESTARTS   AGE
csi-attacher-7cb499df6-9rbd4                1/1     Running             8          14d
csi-attacher-7cb499df6-tmmkn                1/1     Running             5          4d5h
csi-attacher-7cb499df6-xk4hp                0/1     Running             5          13d
csi-provisioner-67846b4b55-67c9z            0/1     Running             11         13d
csi-provisioner-67846b4b55-ld5wc            1/1     Running             5          4d5h
csi-provisioner-67846b4b55-vcz2g            1/1     Running             11         14d
csi-resizer-5cb8df7db9-j54t4                1/1     Running             3          4d5h
csi-resizer-5cb8df7db9-jvmsg                0/1     Running             6          13d
csi-resizer-5cb8df7db9-pfwbh                1/1     Running             8          14d
engine-image-ei-ee18f965-2pvzz              1/1     Running             5          10d
engine-image-ei-ee18f965-5v78m              1/1     Running             7          10d
engine-image-ei-ee18f965-kmfp2              0/1     Running             5          14d
instance-manager-e-0ee5547d                 0/1     Running             0          3d22h
instance-manager-e-898a68ba                 1/1     Running             0          2d22h
instance-manager-e-9a24409b                 1/1     Running             0          2d22h
longhorn-csi-plugin-4gfdx                   0/2     Running             13         14d
longhorn-csi-plugin-6dllz                   2/2     Running             16         14d
longhorn-csi-plugin-7x68l                   2/2     Running             22         10d
longhorn-driver-deployer-658fdf45cc-rkh9s   1/1     Running             7          14d
longhorn-manager-7pc2f                      1/1     Running             6          14d
longhorn-manager-cgjtd                      1/1     Running             7          14d
longhorn-manager-dzw5s                      1/1     Running             7          10d
longhorn-ui-7788d4f485-qd9w8                0/1     Running               2          4d5h

Accessing the Longhorn UI

Remember to replace the content within < > with your info.

Basic Authentication Secret

First, create a secret for Basic Authentication from shell.

1
2
3
4
5
# create ing-auth pass
$ htpasswd -c ./ing-auth admin

# create secret
$ kubectl create secret generic longhorn-auth --from-file ing-auth --namespace=longhorn-system

Or by kubectl apply command.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Secret.longhorn-frontend.yaml

---
# Secrets | Kubernetes
# https://kubernetes.io/docs/concepts/configuration/secret/
apiVersion: v1
kind: Secret
metadata:
  name: longhorn-auth
  namespace: longhorn-system
type: Opaque
data:
  auth: <Your UserName and Password | base64 >

Apply the yaml file.

1
$ kubectl apply -f Secret.longhorn-frontend.yaml

Ingress

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Ingress.longhorn-frontend.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"

    # Basic Authentication - NGINX Ingress Controller
    # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: longhorn-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'

  name: longhorn-frontend
  namespace: longhorn-system
spec:
  rules:
  - host: <Your longhorn-frontend Host>
    http:
      paths:
      - backend:
          serviceName: longhorn-frontend
          servicePort: 80
  tls:
    - secretName: cert-manager-webhook-dnspod-letsencrypt
      hosts:
        - <Your longhorn-frontend Host>

Apply the yaml file.

1
$ kubectl apply -f Ingress.longhorn-frontend.yaml

Then, you can visit Longhorn Front UI with https://

FAQs

instance-manager request too much cpu

1
2
3
$ kubectl describe node node1
  longhorn-system                            instance-manager-e-9cc9f3a3                                   456m (12%)    0 (0%)      0 (0%)           0 (0%)         12d
  longhorn-system                            instance-manager-r-18bb467c                                   456m (12%)    0 (0%)      0 (0%)           0 (0%)         12d

By default instance-manager request 12% CPU on a node. When the number of CPU cores is limit(<=4 cores), the problem of insufficient cpu is prone to occur.

Change guaranteedEngineManagerCPU and guaranteedReplicaManagerCPU value to set CPU limit.

1
2
3
4
5
# values/values.yaml

defaultSettings:
  guaranteedEngineManagerCPU: 5 # 5 means 5% of the total CPU on a node will be allocated to each engine manager pod on this node
  guaranteedReplicaManagerCPU: 5 # 5 means 5% of the total CPU on a node will be allocated to each engine manager pod on this node

Or change engineManagerCPURequest and replicaManagerCPURequest on nodes.longhorn.io resources

1
2
3
4
5
$ kubectl get  nodes.longhorn.io -n longhorn-system
NAMESPACE         NAME                 READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
longhorn-system   node1                True    true              True          33d
longhorn-system   node2                True    true              True          33d
longhorn-system   node3                True    true              True          33d
1
2
3
4
5
# node1.nodes.longhorn.io

sepc:
  engineManagerCPURequest: 100m # 100, 0.1 or 100m
  replicaManagerCPURequest: 100m # 100, 0.1 or 100m

Explanation about guaranteedEngineManagerCPU and guaranteedReplicaManagerCPU.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
- variable: defaultSettings.guaranteedEngineManagerCPU
  label: Guaranteed Engine Manager CPU
  description: "This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each engine manager Pod. For example, 10 means 10% of the total CPU on a node will be allocated to each engine manager pod on this node. This will help maintain engine stability during high node workload.
  In order to prevent unexpected volume engine crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting:
  Guaranteed Engine Manager CPU = The estimated max Longhorn volume engine count on a node * 0.1 / The total allocatable CPUs on the node * 100.
  The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting.
  If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes.
  WARNING:
    - Value 0 means unsetting CPU requests for engine manager pods.
    - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Engine Manager CPU' should not be greater than 40.
    - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then.
    - This global setting will be ignored for a node if the field \"EngineManagerCPURequest\" on the node is set.
    - After this setting is changed, all engine manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES."
  group: "Longhorn Default Settings"
  type: int
  min: 0
  max: 40
  default: 12
- variable: defaultSettings.guaranteedReplicaManagerCPU
  label: Guaranteed Replica Manager CPU
  description: "This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each replica manager Pod. 10 means 10% of the total CPU on a node will be allocated to each replica manager pod on this node. This will help maintain replica stability during high node workload.
  In order to prevent unexpected volume replica crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting:
  Guaranteed Replica Manager CPU = The estimated max Longhorn volume replica count on a node * 0.1 / The total allocatable CPUs on the node * 100.
  The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting.
  If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes.
  WARNING:
    - Value 0 means unsetting CPU requests for replica manager pods.
    - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Replica Manager CPU' should not be greater than 40.
    - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then.
    - This global setting will be ignored for a node if the field \"ReplicaManagerCPURequest\" on the node is set.
    - After this setting is changed, all replica manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES."
  group: "Longhorn Default Settings"
  type: int
  min: 0
  max: 40
  default: 12

See - https://github.com/longhorn/longhorn/blob/e835daf103b121759819caaee6d9cb29483c2f86/chart/questions.yaml to learn more.

References

[1] Install Longhorn with Helm - https://longhorn.io/docs/1.0.2/deploy/install/install-with-helm/

[2] Accessing the Longhorn UI - https://longhorn.io/docs/1.0.2/deploy/accessing-the-ui/

[3] Longhorn - Cloud native distributed block storage for Kubernetes - https://longhorn.io/

[4] Open-iSCSI by open-iscsi - http://www.open-iscsi.com/

[5] Helm - https://helm.sh/

[6] Kubernetes - https://kubernetes.io/