[Kubernetes (K8S)] Helm install Velero to backup, restore and migrate within Kubernetes (K8S)

Posted on Edited on In , , Views: Word count in article: 9.3k Reading time ≈ 8 mins.

Velero

Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

This article is about how to use Helm to install Velero on Kubernetes (K8S).

Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a public cloud platform or on-premises. Velero lets you:

  • Take backups of your cluster and restore in case of loss.

  • Migrate cluster resources to other clusters.

  • Replicate your production cluster to development and testing clusters.

Velero consists of:

  • A server that runs on your cluster

  • A command-line client that runs locally

Prerequisites

  • Kubernetes (K8S)

    Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

    For more information about installing and using Kubernetes (K8s), see the Kubernetes (K8s) Docs.

  • Helm

    Helm is the best way to find, share, and use software built for Kubernetes.

    1
    2
    # Mac OS X
    $ brew install helm

    For more information about installing and using Helm, see the Helm Docs.

  • Amazon S3 Bucket

    Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

    See - https://aws.amazon.com/s3/ to learn more.

Usages

values.yaml

Edit values.yaml and replace content within {{ }} or < >.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
# values.yaml

# helm-charts/values.yaml at main · vmware-tanzu/helm-charts
# https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/values.yaml

# Init containers to add to the Velero deployment's pod spec. At least one plugin provider image is required.
# If the value is a string then it is evaluated as a template.
initContainers:
  # - name: velero-plugin-for-csi
  #   image: velero/velero-plugin-for-csi:v0.2.0
  #   imagePullPolicy: IfNotPresent
  #   volumeMounts:
  #     - mountPath: /target
  #       name: plugins
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.3.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins


# This job is meant primarily for cleaning up CRDs on CI systems.
# Using this on production systems, especially those that have multiple releases of Velero, will be destructive.
cleanUpCRDs: false

##
## Parameters for the `default` BackupStorageLocation and VolumeSnapshotLocation,
## and additional server settings.
##
configuration:
  # Cloud provider being used (e.g. aws, azure, gcp).
  provider: aws

  # Parameters for the `default` BackupStorageLocation. See
  # https://velero.io/docs/v1.6/api-types/backupstoragelocation/
  backupStorageLocation:

    # bucket is the name of the bucket to store backups in. Required.
    bucket: backups

    # prefix is the directory under which all Velero data should be stored within the bucket. Optional.
    prefix: default

    # Additional provider-specific configuration. See link above
    # for details of required/optional fields for your provider.
    config:
      region: {{ .Values.region }}
      s3Url: {{ .Values.s3Url }}

  # Parameters for the `default` VolumeSnapshotLocation. See
  # https://velero.io/docs/v1.6/api-types/volumesnapshotlocation/
  volumeSnapshotLocation:
    # name is the name of the volume snapshot location where snapshots are being taken. Required.
    name: snapshots
    # provider is the name for the volume snapshot provider. If omitted
    # `configuration.provider` will be used instead.
    provider: aws
    # Additional provider-specific configuration. See link above
    # for details of required/optional fields for your provider.
    config:
      region: {{ .Values.region }}

  # Set true for backup all pod volumes without having to apply annotation on the pod when used restic Default: false. Other option: false.
  defaultVolumesToRestic: true

# Info about the secret to be used by the Velero deployment, which
# should contain credentials for the cloud provider IAM account you've
# set up for Velero.
credentials:
  # Whether a secret should be used as the source of IAM account
  # credentials. Set to false if, for example, using kube2iam or
  # kiam to provide IAM credentials for the Velero pod.
  useSecret: true
  # Name of the secret to create if `useSecret` is true and `existingSecret` is empty
  name:
  # Name of a pre-existing secret (if any) in the Velero namespace
  # that should be used to get IAM account credentials. Optional.
  existingSecret:
  # Data to be stored in the Velero secret, if `useSecret` is true and `existingSecret` is empty.
  # As of the current Velero release, Velero only uses one secret key/value at a time.
  # The key must be named `cloud`, and the value corresponds to the entire content of your IAM credentials file.
  # Note that the format will be different for different providers, please check their documentation.
  # Here is a list of documentation for plugins maintained by the Velero team:
  # [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
  # [GCP] https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/README.md
  # [Azure] https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md
  secretContents:
  #  cloud: |
  #    [default]
  #    aws_access_key_id=<REDACTED>
  #    aws_secret_access_key=<REDACTED>
  # additional key/value pairs to be used as environment variables such as "DIGITALOCEAN_TOKEN: <your-key>". Values will be stored in the secret.
    cloud: |
      [default]
      aws_access_key_id = <Your Cloud Access Key>
      aws_secret_access_key = <Your Cloud Access Secret>

# Whether to deploy the restic daemonset.
deployRestic: true

restic:
  privileged: true

Install Release

Helm install Velero into Velero namespace.

1
2
3
4
5
6
7
8
9
10
11
# crate namespace:
$ kubectl create namespace velero

# Add the vmware-tanzu Helm repository:
$ helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

# Update your local Helm chart repository cache:
$ helm repo update

# To install Helm chart:
$ helm install velero vmware-tanzu/velero -f values.yaml

See pods about velero.

1
2
3
4
5
6
$ kubectl get pods -n velero
NAME                      READY   STATUS    RESTARTS      AGE
restic-c2m2x              1/1     Running   0             8h
restic-tmx6n              1/1     Running   0             8h
restic-zs6lw              1/1     Running   1 (11m ago)   8h
velero-645b5c7c75-4bwlh   1/1     Running   0             8h

Examples

Install Velero CLI

Option 1: MacOS - Homebrew

1
2
# On macOS, you can use Homebrew to install the velero client:
$ brew install velero

Option 2: GitHub release

    1. Download the latest release’s tarball for your client platform.
    1. Extract the tarball:
1
$ tar -xvf <RELEASE-TARBALL-NAME>.tar.gz
    1. Move the extracted velero binary to somewhere in your $PATH (/usr/local/bin for most users).

See Install the CLI Velero Docs - Basic Install - https://velero.io/docs/v1.7/basic-install/#install-the-cli to learn more.

Deploy examples/nginx-app/with-pv

First, deploy nginx-app into namespace nginx-example

1
$ kubectl apply -f examples/nginx-app/with-pv.yaml

Run backup

Option 1 Create backup via yaml

Here is a sample Backup object with each of the fields documented manifests/Backup-nginx-example.yaml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# manifests/Backup-nginx-example.yaml

# Standard Kubernetes API Version declaration. Required.
apiVersion: velero.io/v1
# Standard Kubernetes Kind declaration. Required.
kind: Backup
# Standard Kubernetes metadata. Required.
metadata:
  # Backup name. May be any valid Kubernetes object name. Required.
  name: nginx-example
  # Backup namespace. Must be the namespace of the Velero server. Required.
  namespace: velero
# Parameters about the backup. Required.
spec:
  # Array of namespaces to include in the backup. If unspecified, all namespaces are included.
  # Optional.
  includedNamespaces:
  - nginx-example

See - https://velero.io/docs/v1.7/api-types/backup/ to learn more.

Apply manifests/Backup-nginx-example.yaml manifest file.

1
$ kubectl apply -f manifests/Backup-nginx-example.yaml

Option 2 Create backup via Velero CLI

1
$ velero backup create nginx-backup --include-namespaces nginx-example

Check Velero backup status

1
2
3
4
5
$ velero backup get nginx-example

$ velero backup describe nginx-example --details

$ velero backup logs nginx-example

Restore

Simulate a disaster:

1
$ kubectl delete namespaces nginx-example

Before continuing to the next step, check your cloud provider to confirm that the disk no longer exists.

Restore your lost resources:

1
velero restore create --from-backup nginx-backup

All resources of nginx-backup will restore.

FAQs

unable to locate ObjectStore plugin named velero.io/aws

Enable velero-plugin-for-aws for initContainers.

1
2
3
4
5
6
7
8
9
10
11
# values.yaml

# Init containers to add to the Velero deployment's pod spec. At least one plugin provider image is required.
# If the value is a string then it is evaluated as a template.
initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.3.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

pod volume backup failed: error getting volume path on host

1
time="2021-12-14T10:14:15Z" level=error msg="Error backing up item" backup=velero/argo-cd error="pod volume backup failed: error getting volume path on host: expected one matching path, got 0" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:184" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:435" name=argo-cd-argocd-server-c8f945b59-j7b4c

Velero’s restic daemonset uses a hostPath mount of /var/lib/kubelet/pods to get access to pods’ volume mounts. If a pod is using a hostPath mount, it won’t be mounted under that directory, so Velero isn’t able to access it.

See [FAQ] Why does Velero/restic not support hostPath volumes? · Discussion #3378 · vmware-tanzu/velero - https://github.com/vmware-tanzu/velero/discussions/3378 to learn more.

References

[1] VMware Tanzu Helm Repository | VMware Tanzu Helm Charts - https://vmware-tanzu.github.io/helm-charts/

[2] Velero - https://velero.io/

[3] vmware-tanzu/velero: Backup and migrate Kubernetes applications and their persistent volumes - https://github.com/vmware-tanzu/velero

[4] Kubernetes Getting Started | Pulumi - https://www.pulumi.com/docs/get-started/kubernetes/

[5] Kubernetes - https://kubernetes.io/

[6] Helm - https://helm.sh/

[7] Amazon S3 Bucket - https://aws.amazon.com/s3/