Longhorn is a lightweight, reliable and easy-to-use Cloud native distributed block storage system for Kubernetes.
Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation.
With Longhorn, you can:
Use Longhorn volumes as persistent storage for the distributed stateful applications in your Kubernetes cluster
Partition your block storage into Longhorn volumes so that you can use Kubernetes volumes with or without a cloud provider
Replicate block storage across multiple nodes and data centers to increase availability
Store backup data in external storage such as NFS or AWS S3
Create cross-cluster disaster recovery volumes so that data from a primary Kubernetes cluster can be quickly recovered from backup in a second Kubernetes cluster
Schedule recurring snapshots of a volume, and schedule recurring backups to NFS or S3-compatible secondary storage
Restore volumes from backup
Upgrade Longhorn without disrupting persistent volumes
Manipulate Longhorn resources with kubectl
This article is about how to use Helm to deploy Longhorn on Kubernetes (K8S).
Pulumi is a modern infrastructure-as-code platform that allows you to use common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources.
open-iscsi is installed, and the iscsid daemon is running on all the nodes.
The Open-iSCSI project provides a high-performance, transport independent, implementation of RFC 3720 iSCSI for Linux.
You will get an error message without open-iscsi.
1
level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host"
For help installing open-iscsi:
1 2 3 4 5
# For Debian and Ubuntu, use this command: $ sudo apt-get install -y open-iscsi # For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command: $ sudo yum install -y iscsi-initiator-utils
1 2 3
# start and enable iscsid.service or reboot $ sudo systemctl start iscsid && sudo systemctl enable iscsid $ sudo systemctl status iscsid
Usage
Pulumi New
Create the workspace directory.
1 2 3
$ mkdir -p col-example-pulumi-typescript-longhorn $ cd col-example-pulumi-typescript-longhorn
Pulumi login into local file system.
1 2 3
$ pulumi login file://. Logged in to cloudolife as cloudolife (file://.) or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
Pulumi new a project with kubernetes-typescript SDK.
1
$ pulumi new kubernetes-typescript
The above command will create some files within the current directory.
Install js-yaml package to load and parse yaml file.
1
$ npm i js-yaml
Pulumi Configuration
Configure Kubernetes
By default, Pulumi will look for a kubeconfig file in the following locations, just like kubectl:
The environment variable: $KUBECONFIG,
Or in current user’s default kubeconfig directory: ~/.kube/config
If the kubeconfig file is not in either of these locations, Pulumi will not find it, and it will fail to authenticate against the cluster. Set one of these locations to a valid kubeconfig file, if you have not done so already.
Configure Values.yaml
Edit values.yaml and replace content within {{ }}.
# longhorn/values.yaml at master · longhorn/longhorn # https://github.com/longhorn/longhorn/blob/master/chart/values.yaml
# persistence: # defaultClass: true
ingress: ## Set to true to enable ingress record generation enabled:true
host: {{ .Values.host }}
## Set this to true in order to enable TLS on the ingress record ## A side effect of this will be that the backend service will be connected at port 443 tls:true
## If TLS is set to true, you must declare what secret will store the key/certificate for TLS tlsSecret: {{ .Values.ingress.tlsSecret }}
## Ingress annotations done as key:value pairs ## If you're using kube-lego, you will want to add: ## kubernetes.io/tls-acme: true ## ## For a full list of possible ingress annotations, please see ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/annotations.md ## ## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set annotations: kubernetes.io/ingress.class:nginx
# Basic Authentication - NGINX Ingress Controller # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/ # type of authentication nginx.ingress.kubernetes.io/auth-type:basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret:longhorn-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm:'Authentication Required - foo'
Verify that any required CRDs have been created: no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.
Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.
All existing persisted objects are accessible via the new API
Notable changes:
- spec.backend is renamed to spec.defaultBackend
- The backend serviceName field is renamed to service.name
- Numeric backend servicePort fields are renamed to service.port.number
- String backend servicePort fields are renamed to service.port.name
- pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.
By default instance-manager request 12% CPU on a node. When the number of CPU cores is limit(<=4 cores), the problem of insufficient cpu is prone to occur.
Change guaranteedEngineManagerCPU and guaranteedReplicaManagerCPU value to set CPU limit.
1 2 3 4 5
# values/values.yaml
defaultSettings: guaranteedEngineManagerCPU:5# 5 means 5% of the total CPU on a node will be allocated to each engine manager pod on this node guaranteedReplicaManagerCPU:5# 5 means 5% of the total CPU on a node will be allocated to each engine manager pod on this node
Or change engineManagerCPURequest and replicaManagerCPURequest on nodes.longhorn.io resources
1 2 3 4 5
$ kubectl get nodes.longhorn.io -n longhorn-system NAMESPACE NAME READY ALLOWSCHEDULING SCHEDULABLE AGE longhorn-system node1 True true True 33d longhorn-system node2 True true True 33d longhorn-system node3 True true True 33d
1 2 3 4 5
# node1.nodes.longhorn.io
sepc: engineManagerCPURequest: 100m # 100, 0.1 or 100m replicaManagerCPURequest: 100m # 100, 0.1 or 100m
Explanation about guaranteedEngineManagerCPU and guaranteedReplicaManagerCPU.
-variable:defaultSettings.guaranteedEngineManagerCPU label:GuaranteedEngineManagerCPU description:"This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each engine manager Pod. For example, 10 means 10% of the total CPU on a node will be allocated to each engine manager pod on this node. This will help maintain engine stability during high node workload. In order to prevent unexpected volume engine crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting: Guaranteed Engine Manager CPU = The estimated max Longhorn volume engine count on a node * 0.1 / The total allocatable CPUs on the node * 100. The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting. If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes. WARNING: - Value 0 means unsetting CPU requests for engine manager pods. - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Engine Manager CPU' should not be greater than 40. - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then. - This global setting will be ignored for a node if the field \"EngineManagerCPURequest\" on the node is set. - After this setting is changed, all engine manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES." group:"Longhorn Default Settings" type:int min:0 max:40 default:12 -variable:defaultSettings.guaranteedReplicaManagerCPU label:GuaranteedReplicaManagerCPU description:"This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each replica manager Pod. 10 means 10% of the total CPU on a node will be allocated to each replica manager pod on this node. This will help maintain replica stability during high node workload. In order to prevent unexpected volume replica crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting: Guaranteed Replica Manager CPU = The estimated max Longhorn volume replica count on a node * 0.1 / The total allocatable CPUs on the node * 100. The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting. If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes. WARNING: - Value 0 means unsetting CPU requests for replica manager pods. - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Replica Manager CPU' should not be greater than 40. - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then. - This global setting will be ignored for a node if the field \"ReplicaManagerCPURequest\" on the node is set. - After this setting is changed, all replica manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES." group:"Longhorn Default Settings" type:int min:0 max:40 default:12