[Infrastructure as Code (IaC) Pulumi] Use Pulumi kubernetes (K8S) Helm Chart to deploy Elastic Kibana

Posted on 2021-09-11 Edited on 2024-09-12 In Infrastructure as Code (IaC) , Cloud Native , Kubernetes (K8S) , Pulumi Views: Word count in article: 8.3k Reading time ≈ 8 mins.

Elastic Kibana

Elastic Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.

This article is about how to use Pulumi, kubernetes (K8S) provider, Helm Chart and TypeScript SDK to deploy Elastic Kibana within Kubernetes (K8S).

Prerequisites

Kubernetes (K8S)

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

See Getting started | Kubernetes - https://kubernetes.io/docs/setup/ to leanr more.
Pulumi - Modern Infrastructure as Code - https://www.pulumi.com/

Pulumi is a modern infrastructure-as-code platform that allows you to use common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources.

Install the Pulumi - https://www.pulumi.com/ CLI.
1
2
# Mac OS X
$ brew install pulumi
See Download and Install | Pulumi - https://www.pulumi.com/docs/get-started/install/ to learn more about others OS.
Node.js - https://nodejs.org/en/

Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine.

Install Node.js - https://nodejs.org/en/ CLI.
1
2
# Mac OS X
$ brew install node
See Node.js - https://nodejs.org/en/ to learn more about others OS.
Elasticsearch - https://www.elastic.co/elasticsearch/

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

Usage

Pulumi New

Create the workspace directory.

1
2
3

$ mkdir -p col-example-pulumi-typescript-kibana

$ cd col-example-pulumi-typescript-kibana

Pulumi login into local file system.

1
2
3

$ pulumi login file://.
Logged in to cloudolife as cloudolife (file://.)
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.

Pulumi new a project with kubernetes-typescript SDK.

1	$ pulumi new kubernetes-typescript

The above command will create some files within the current directory.

tree . -L 1
.
├── node_modules/
├── package.json
├── package.json.lock
├── Pulumi.dev.yaml
├── Pulumi.yaml
└── main.ts

Install js-yaml package to load and parse yaml file.

1	$ npm i js-yaml

Pulumi Configuration

Configure Kubernetes

By default, Pulumi will look for a kubeconfig file in the following locations, just like kubectl:

The environment variable: $KUBECONFIG,
Or in current user’s default kubeconfig directory: ~/.kube/config

If the kubeconfig file is not in either of these locations, Pulumi will not find it, and it will fail to authenticate against the cluster. Set one of these locations to a valid kubeconfig file, if you have not done so already.

Generate a Admin and password Auth.

1
2
3

$ htpasswd -c ./ing-auth admin

$ cat ./ing-auth | base64

Remember the base64 output as the next Secret data.

manifests

Edit manifests/Secret.yaml and replace content within {{ }}.

# manifests/Secret.yaml

---
# Secrets | Kubernetes
# https://kubernetes.io/docs/concepts/configuration/secret/
apiVersion: v1
kind: Secret
metadata:
  name: elastic-kibana-auth
  namespace: elastic
type: Opaque
data:
  # htpasswd -c ing-auth admin
  # cat ing-auth | base64
  auth: {{ .Values.auth }}

Configure Values.yaml

Edit values.yaml and replace content within {{ }}.

# values.yaml

# helm-charts/values.yaml at master · elastic/helm-charts
# https://github.com/elastic/helm-charts/blob/master/kibana/values.yaml

ingress:
  enabled: true

  annotations:
    kubernetes.io/ingress.class: nginx

    # Basic Authentication - NGINX Ingress Controller
    # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: elastic-kibana-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'

  hosts:
    - host: {{ .Values.host }}

  tls:
    - secretName: {{ .Values.tls.secretName }}

main.ts

// main.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameElastic = "elastic"

// kubernetes.core/v1.Namespace | Pulumi
// https://www.pulumi.com/docs/reference/pkg/kubernetes/core/v1/namespace/
const namespaceElastic = new k8s.core.v1.Namespace(nameElastic, {
    metadata: {
        name: nameElastic,
    },
})

const configGroup = new k8s.yaml.ConfigGroup(name, {
    files: ["manifests/*.yaml"],
}, options);

const values = yaml.safeLoad(fs.readFileSync("./values.yaml", 'utf8'))

const charNameKibana = "kibana"

const charKibana = new k8s.helm.v3.Chart(charNameKibana, {
    chart: charNameKibana,
    version: "7.15.0",
    fetchOpts:{
        repo: "https://helm.elastic.co",
    },
    values: values,
    namespace: namespaceElastic.metadata.name,
})

Pulumi Up

Run pulumi up to create the namespace and pods.

1	$ pulumi up

See pods about kibana.

1
2
3

$ kubectl get pods -n elastic
NAME                                     READY   STATUS    RESTARTS        AGE
kibana-kibana-56689685dc-qcl4g           1/1     Running   9 (15m ago)     3d12h

Pulumi Destroy

Destroy all resources created by Pulumi.

1	$ pulumi destroy

FAQs

no matches for kind “Ingress” in version “networking.k8s.io/v1beta1” since Kubernets (K8S) v1.22

1	Verify that any required CRDs have been created: no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"

The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.

Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.

All existing persisted objects are accessible via the new API

Notable changes:

  - spec.backend is renamed to spec.defaultBackend
  
  - The backend serviceName field is renamed to service.name

  - Numeric backend servicePort fields are renamed to service.port.number

  - String backend servicePort fields are renamed to service.port.name
  
  - pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.

See Deprecated API Migration Guide | Kubernetes - https://kubernetes.io/docs/reference/using-api/deprecation-guide/ to learn more.

First, create Ingress.yaml with new Ingress with networking.k8s.io/v1 API version.

# manifests/Ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # Basic Authentication - NGINX Ingress Controller
    # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: elastic-kibana-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'

  name: kibana
  namespace: elastic
spec:
  rules:
    - host: {{ .Values.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kibana-kibana
                port:
                  number: 5601
  tls:
    - secretName: {{ .Values.tls.secretName }}

See Ingress | Kubernetes - https://kubernetes.io/docs/concepts/services-networking/ingress/ to leanr more.

Then, run kubectl apply command.

$ kubectl apply -f Ingress.yaml

$ kubectl get ingress -n elastic
NAME     CLASS    HOSTS                                       ADDRESS        PORTS     AGE
kibana   <none>   elastic-kibana.cloudolife.com   10.233.59.32   80, 443   8d

Now, you can visite Grafana Loki with https://{{ .Values.host }}.