[Infrastructure as Code (IaC) Pulumi] Use Pulumi kubernetes (K8S) Helm Chart to deploy Elastic Stack (Elasticsearch, APM Server, Kibana, Fluent Bit or Fluentd)

Posted on 2021-11-20 Edited on 2024-09-12 In Infrastructure as Code (IaC) , Cloud Native , Kubernetes (K8S) , Pulumi Views: Word count in article: 11k Reading time ≈ 10 mins.

Elastic APM

Elastic Stack includes Elasticsearch, APM Server, Kibana, Fluent Bit or Fluentd etc.

This article is about how to use Pulumi, kubernetes (K8S) provider, Helm Chart and TypeScript SDK to deploy Elastic Stack (Elasticsearch, APM Server, Kibana, Fluent Bit or Fluentd) within Kubernetes (K8S).

Elasticsearch: The Official Distributed Search & Analytics Engine | Elastic - https://www.elastic.co/elasticsearch/

Elasticsearch is the distributed, RESTful search and analytics engine at the heart of the Elastic Stack. You can use Elasticsearch to store, search, and manage data for Logs, Metrics, A search backend, Application monitoring, Endpoint security and more!
Application Performance Monitoring (APM) with Elasticsearch | Elastic - https://www.elastic.co/apm/

Elastic APM is the free and open application performance monitoring. It monitor, detect, and diagnose complex application performance issues to maintain an expected level of service.
elastic/kibana: Your window into the Elastic Stack - https://github.com/elastic/kibana

Elastic Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.
Fluent Bit or Fluentd
- (Recommend) Fluent Bit - https://fluentbit.io/
  
  Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters and send them to multiple destinations. It’s the preferred choice for containerized environments like Kubernetes.
- fluent/helm-charts: Helm Charts for Fluentd and Fluent Bit - https://github.com/fluent/helm-charts
  
  Fluentd is an open source data collector for unified logging layer.

Prerequisites

Kubernetes (K8S)

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

See Getting started | Kubernetes - https://kubernetes.io/docs/setup/ to leanr more.
Pulumi - Modern Infrastructure as Code - https://www.pulumi.com/

Pulumi is a modern infrastructure-as-code platform that allows you to use common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources.

Install the Pulumi - https://www.pulumi.com/ CLI.
1
2
# Mac OS X
$ brew install pulumi
See Download and Install | Pulumi - https://www.pulumi.com/docs/get-started/install/ to learn more about others OS.
Node.js - https://nodejs.org/en/

Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine.

Install Node.js - https://nodejs.org/en/ CLI.
1
2
# Mac OS X
$ brew install node
See Node.js - https://nodejs.org/en/ to learn more about others OS.

Usage

Pulumi New

Create the workspace directory.

1
2
3

$ mkdir -p col-example-pulumi-typescript-elastic-stack

$ cd col-example-pulumi-typescript-elastic-stack

Pulumi login into local file system.

1
2
3

$ pulumi login file://.
Logged in to cloudolife as cloudolife (file://.)
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.

Pulumi new a project with kubernetes-typescript SDK.

1	$ pulumi new kubernetes-typescript

The above command will create some files within the current directory.

tree . -L 1
.
├── node_modules/
├── package.json
├── package.json.lock
├── Pulumi.dev.yaml
├── Pulumi.yaml
└── main.ts

Install js-yaml package to load and parse yaml file.

1	$ npm i js-yaml

Pulumi Configuration

Configure Kubernetes

By default, Pulumi will look for a kubeconfig file in the following locations, just like kubectl:

The environment variable: $KUBECONFIG,
Or in current user’s default kubeconfig directory: ~/.kube/config

If the kubeconfig file is not in either of these locations, Pulumi will not find it, and it will fail to authenticate against the cluster. Set one of these locations to a valid kubeconfig file, if you have not done so already.

Elasticsearch

values.yaml

Edit elasticsearch/values.yaml and replace content within {{ }}.

# values.yaml

# helm-charts/values.yaml at main · elastic/helm-charts
# https://github.com/elastic/helm-charts/blob/main/elasticsearch/values.yaml

index.ts

Edit elasticsearch/index.ts

// elasticsearch/index.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameElasticStack = "elastic-stack"

// kubernetes.core/v1.Namespace | Pulumi
// https://www.pulumi.com/docs/reference/pkg/kubernetes/core/v1/namespace/
const namespaceElasticStack = new k8s.core.v1.Namespace(nameElasticStack, {
    metadata: {
        name: nameElasticStack,
    },
})

const values = yaml.safeLoad(fs.readFileSync("./elasticsearch/values.yaml", 'utf8'))

const charNameElasticsearch = "elasticsearch"

export const charElasticsearch = new k8s.helm.v3.Chart(charNameElasticsearch, {
    chart: charNameElasticsearch,
    version: "7.15.0",
    fetchOpts:{
        repo: "https://helm.elastic.co",
    },
    values: values,
    namespace: namespaceElasticStack.metadata.name,
})

Elastic APM Server

values.yaml

Edit apm-server/values.yaml and replace content within {{ }}.

# apm-server/values.yaml

# helm-charts/values.yaml at master · elastic/helm-charts
# https://github.com/elastic/helm-charts/blob/master/apm-server/values.yaml

# Allows you to add config files
apmConfig:
  apm-server.yml: |
    apm-server:
      host: "0.0.0.0:8200"
      
    output.elasticsearch:
      hosts: {{ .Values.apm-server.elasticsearch_url }}

index.ts

Edit elasticsearch/index.ts

// elasticsearch/index.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameElastic = "elastic"

// kubernetes.core/v1.Namespace | Pulumi
// https://www.pulumi.com/docs/reference/pkg/kubernetes/core/v1/namespace/
const namespaceElastic = new k8s.core.v1.Namespace(nameElastic, {
    metadata: {
        name: nameElastic,
    },
})

const values = yaml.safeLoad(fs.readFileSync("./apm-server/values.yaml", 'utf8'))

const charNameApmServer = "apm-server"

export const charApmServer = new k8s.helm.v3.Chart(charNameApmServer, {
    chart: charNameApmServer,
    version: "7.15.0",
    fetchOpts:{
        repo: "https://helm.elastic.co",
    },
    values: values,
    namespace: namespaceElastic.metadata.name,
})

Elastic Kibana

Generate a Admin and password Auth.

1
2
3

$ htpasswd -c ./ing-auth admin

$ cat ./ing-auth | base64

Remember the base64 output as the next Secret data.

manifests

Edit manifests/kibana/Secret.yaml and replace content within {{ }}.

# manifests/kibana/Secret.yaml

---
# Secrets | Kubernetes
# https://kubernetes.io/docs/concepts/configuration/secret/
apiVersion: v1
kind: Secret
metadata:
  name: elastic-kibana-auth
  namespace: elastic
type: Opaque
data:
  # htpasswd -c ing-auth admin
  # cat ing-auth | base64
  auth: {{ .Values.auth }}

values.yaml

Edit values.yaml and replace content within {{ }}.

# kibana/values.yaml

# helm-charts/values.yaml at master · elastic/helm-charts
# https://github.com/elastic/helm-charts/blob/master/kibana/values.yaml

ingress:
  enabled: true

  annotations:
    kubernetes.io/ingress.class: nginx

    # Basic Authentication - NGINX Ingress Controller
    # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: elastic-kibana-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'

  hosts:
    - host: {{ .Values.host }}

  tls:
    - secretName: {{ .Values.tls.secretName }}

index.ts

Edit ./kibana/index.ts

// kibana/index.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const configGroup = new k8s.yaml.ConfigGroup(name, {
    files: ["kibana/manifests/*.yaml"],
}, options);

const yaml = require('js-yaml');
const fs   = require('fs');

const values = yaml.safeLoad(fs.readFileSync("./kibana/values.yaml", 'utf8'))

const charNameKibana = "kibana"

export const charKibana = new k8s.helm.v3.Chart(charNameKibana, {
    chart: charNameKibana,
    version: "7.15.0",
    fetchOpts:{
        repo: "https://helm.elastic.co",
    },
    values: values,
    namespace: "elastic",
})

Fluent Bit

values.yaml

Edit fluent-bit/values.yaml and replace content within {{ }}.

# fluent-bit/values.yaml

# helm-charts/values.yaml at main · fluent/helm-charts
# https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml

index.ts

Edit ./kibana/index.ts

// kibana/index.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameElastic = "elastic"

const values = yaml.safeLoad(fs.readFileSync("./fluent-bit/values.yaml", 'utf8'))

const charNameFluentBit = "fluent-bit"

export const charFluentBit = new k8s.helm.v3.Chart(charNameFluentBit, {
    chart: charNameFluentBit,
    version: "0.19.7",
    fetchOpts:{
        repo: "https://fluent.github.io/helm-charts",
    },
    values: values,
    namespace: elastic,
})

Elastic Stack

main.ts

Edit ./main.ts

// main.ts

import * from "./elasticsearch";

import * from "./apm-server";

import * from "./kibana";

import * from "./fluent-bit";

Check all files.

tree . -L 1
.
├── apm-server
│   ├── index.ts
│   └── values
│       └── values.yaml
├── elasticsearch
│   ├── index.ts
│   └── values
│       └── values.yaml
├── fluent-bit
│   ├── index.ts
│   └── values
│       └── values.yaml
├── kibana
│   ├── index.ts
│   ├── manifests
│   │   └── Secret.yaml
│   └── values
│       └── values.yaml
└── x509-certificate-exporter
    ├── index.ts
    └── values
        └── values.yaml
├── package.json
├── package.json.lock
├── Pulumi.dev.yaml
├── Pulumi.yaml
└── main.ts

Pulumi Up

Run pulumi up to create the namespace and pods.

1	$ pulumi up

See pods about apm-server.

$ kubectl get pods -n elastic
NAME                                     READY   STATUS    RESTARTS   AGE
apm-server-apm-server-86c7f664b4-k4dpv   1/1     Running   0          16h
elasticsearch-master-0                   1/1     Running   0          16h
elasticsearch-master-1                   1/1     Running   0          16h
elasticsearch-master-2                   1/1     Running   0          16h
fluent-bit-5twrq                         1/1     Running   0          16h
fluent-bit-b29zc                         1/1     Running   0          16h
fluent-bit-gnmqw                         1/1     Running   0          16h
kibana-kibana-56689685dc-gvcft           1/1     Running   0          16h