[Infrastructure as Code (IaC) Pulumi] Use Pulumi kubernetes (K8S) Helm Chart to deploy Grafana Loki Stack

Posted on 2021-09-25 Edited on 2023-09-02 In Infrastructure as Code (IaC) , Cloud Native , Kubernetes (K8S) , Pulumi Views: Word count in article: 12k Reading time ≈ 11 mins.

Grafana Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

This article is about how to use Pulumi, kubernetes (K8S) provider, Helm Chart and TypeScript SDK to deploy Grafana Loki Stack within Kubernetes (K8S).

Loki is a datastore optimized for efficiently holding log data. The efficient indexing of log data distinguishes Loki from other logging systems. Unlike other logging systems, a Loki index is built from labels, leaving the original log message unindexed.

Loki features

Efficient memory usage for indexing the logs

By indexing on a set of labels, the index can be significantly smaller than other log aggregation products. Less memory makes it less expensive to operate.
Multi-tenancy

Loki allows multiple tenants to utilize a single Loki instance. The data of distinct tenants is completely isolated from other tentants. Multi-tenancy is configured by assigning a tenant ID in the agent.
LogQL, Loki’s query language

Users of the Prometheus query language, PromQL, will find LogQL familiar and flexible for generating queries against the logs. The language also facilitates the generation of metrics from log data, a powerful feature that goes well beyond log aggregation.
Scalability

Loki works well at small scale. In single process mode, all required microservices run in one process. Single process mode is great for testing Loki, running it locally, or running it at a small scale.

Loki is also designed to scale out for large scale installations. Each of the Loki’s microservice components can be broken out into separate processes, and configuration permits individual scaling of the components.
Flexibility

Many agents (clients) have plugin support. This allows a current observability structure to add Loki as their log aggregation tool without needing to switch existing portions of the observability stack.
Grafana integration

Loki seamlessly integrates with Grafana, providing a complete observability stack.

Prerequisites

Kubernetes (K8S)

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

See Getting started | Kubernetes - https://kubernetes.io/docs/setup/ to leanr more.
Pulumi - Modern Infrastructure as Code - https://www.pulumi.com/

Pulumi is a modern infrastructure-as-code platform that allows you to use common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources.

Install the Pulumi - https://www.pulumi.com/ CLI.
1
2
# Mac OS X
$ brew install pulumi
See Download and Install | Pulumi - https://www.pulumi.com/docs/get-started/install/ to learn more about others OS.
Node.js - https://nodejs.org/en/

Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine.

Install Node.js - https://nodejs.org/en/ CLI.
1
2
# Mac OS X
$ brew install node
See Node.js - https://nodejs.org/en/ to learn more about others OS.

Usage

Pulumi New

Create the workspace directory.

1
2
3

$ mkdir -p col-example-pulumi-typescript-loki-stack

$ cd col-example-pulumi-typescript-loki-stack

Pulumi login into local file system.

1
2
3

$ pulumi login file://.
Logged in to cloudolife as cloudolife (file://.)
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.

Pulumi new a project with kubernetes-typescript SDK.

1	$ pulumi new kubernetes-typescript

The above command will create some files within the current directory.

tree . -L 1
.
├── node_modules/
├── package.json
├── package.json.lock
├── Pulumi.dev.yaml
├── Pulumi.yaml
└── main.ts

Install js-yaml package to load and parse yaml file.

1	$ npm i js-yaml

Pulumi Configuration

Configure Kubernetes

By default, Pulumi will look for a kubeconfig file in the following locations, just like kubectl:

The environment variable: $KUBECONFIG,
Or in current user’s default kubeconfig directory: ~/.kube/config

If the kubeconfig file is not in either of these locations, Pulumi will not find it, and it will fail to authenticate against the cluster. Set one of these locations to a valid kubeconfig file, if you have not done so already.

Configure Values.yaml

Edit values.yaml and replace content within {{ }}.

You should have a exist StorageClass in order to persistent data for Granfa and Granfa Loki.

# loki/values.yaml at master · grafana/loki
# https://github.com/grafana/loki/blob/master/production/helm/loki-stack/values.yaml

# Deploy Loki Stack (Loki, Promtail, Grafana, Prometheus) with persistent volume claim
# https://grafana.com/docs/loki/latest/installation/helm/#deploy-loki-stack-loki-promtail-grafana-prometheus-with-persistent-volume-claim

# helm-charts/values.yaml at main · grafana/helm-charts
# https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml
grafana:
  enabled: true

  ingress:
    enabled: true
    # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
    # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
    ingressClassName: nginx
    # Values can be templated
    annotations:
      kubernetes.io/ingress.class: nginx

    hosts:
      - {{ .Values.host }}

    tls: 
      - secretName: {{ .Values.tls.secretName }}

  # Administrator credentials when not using an existing secret (see below)
  adminUser: admin
  adminPassword: "{{ .Values.adminPassword }}"

  ## Enable persistence using Persistent Volume Claims
  ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
  ##
  persistence:
  
    enabled: enable

prometheus:
  enabled: true

  alertmanager:
    persistentVolume:
      enabled: true
      
  server:
    persistentVolume:
      enabled: true

  ## Deploy node exporter as a daemonset to all nodes
  ##
  nodeExporter:
    enabled: true

    extraHostPathMounts:
      - name: dev
        hostPath: /dev
        mountPath: /host/dev
        readOnly: true

      # - name: rootfs
      #   hostPath: /
      #   mountPath: /rootfs
      #   readOnly: true

# helm-charts/values.yaml at main · grafana/helm-charts
# https://github.com/grafana/helm-charts/blob/main/charts/loki/values.yaml      
loki:

  ingress:
    enabled: true
    annotations: 
      kubernetes.io/ingress.class: nginx
    hosts:
      - host: chart-example.local
    tls: 
      - secretName: chart-example-tls

  persistence:
    enabled: true
    
    storageClassName: {{ .Values.loki.persistence.storageClassName }}
    size: 50Gi

See and modify main.ts file.

// main.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameLokiStack = "loki-stack"

// kubernetes.core/v1.Namespace | Pulumi
// https://www.pulumi.com/docs/reference/pkg/kubernetes/core/v1/namespace/
const namespaceLokiStack = new k8s.core.v1.Namespace(nameLokiStack, {
    metadata: {
        name: nameLokiStack,
    },
})

const values = yaml.safeLoad(fs.readFileSync("./values.yaml", 'utf8'))

const charNameLokiStack = "loki-stack"

// https://grafana.com/oss/loki/
// Grafana Loki | Grafana Labs

// grafana/loki: Like Prometheus, but for logs.
// https://github.com/grafana/loki

// loki-stack 2.4.1 · grafana/grafana
// https://artifacthub.io/packages/helm/grafana/loki-stack
const charLokiStack = new k8s.helm.v3.Chart(charNameLokiStack, {
    chart: charNameLokiStack,
    version: "2.4.1",
    fetchOpts:{
        repo: "https://grafana.github.io/loki/charts",
    },
    namespace: namespaceLokiStack.metadata.name,
    values: values,

    transformations: [
                (obj: any, opts: pulumi.CustomResourceOptions) => {
                    // console.log(obj.kind, obj.apiVersion)
                    // Migrate manifests and API clients to use the `networking.k8s.io/v1` API version, available since v1.19.
                    if (obj.apiVersion === "rbac.authorization.k8s.io/v1beta1") {
                        obj.apiVersion = "rbac.authorization.k8s.io/v1"
                    }
                },
            ],
})

Pulumi Up

Run pulumi up to create the namespace and pods.

1	$ pulumi up

See pods about loki-stack.

$ kubectl get pods -n loki-stack
NAME                                                  READY   STATUS    RESTARTS   AGE
loki-stack-0                                          1/1     Running   0          68m
loki-stack-grafana-5cbcd89778-xss94                   1/1     Running   0          68m
loki-stack-kube-state-metrics-b7f44997c-j7wss         1/1     Running   0          68m
loki-stack-prometheus-alertmanager-6b464466f5-fjd6c   2/2     Running   0          68m
loki-stack-prometheus-node-exporter-9g4f5             1/1     Running   0          68m
loki-stack-prometheus-node-exporter-mrs65             1/1     Running   0          68m
loki-stack-prometheus-node-exporter-w9zxb             1/1     Running   0          68m
loki-stack-prometheus-pushgateway-5ff5849598-dwnmg    1/1     Running   0          68m
loki-stack-prometheus-server-9d48df6c5-pbx9k          2/2     Running   0          68m
loki-stack-promtail-kmxvs                             1/1     Running   0          68m
loki-stack-promtail-t22jr                             1/1     Running   0          68m
loki-stack-promtail-xbd75                             1/1     Running   0          68m

Then, you can visite Grafana Loki with https://{{ .Values.host }}.

FAQs

no matches for kind “ClusterRole”, “ClusterRoleBinding”, “Role” in version “rbac.authorization.k8s.io/v1beta1” since Kubernets (K8S) v1.22

The rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.

Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8.
All existing persisted objects are accessible via the new APIs
No notable changes

no matches for kind “Ingress” in version “networking.k8s.io/v1beta1” since Kubernets (K8S) v1.22

1	Verify that any required CRDs have been created: no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"

The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.

Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.

All existing persisted objects are accessible via the new API

Notable changes:

  - spec.backend is renamed to spec.defaultBackend
  
  - The backend serviceName field is renamed to service.name

  - Numeric backend servicePort fields are renamed to service.port.number

  - String backend servicePort fields are renamed to service.port.name
  
  - pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.

See Deprecated API Migration Guide | Kubernetes - https://kubernetes.io/docs/reference/using-api/deprecation-guide/ to learn more.

First, create new Ingress with networking.k8s.io/v1 API version.

# Ingress.loki-stack-grafana.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  name: loki-stack-grafana
  namespace: loki-stack
spec:
  rules:
    - host: {{ .Value.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: loki-stack-grafana
                port:
                  number: 80
  tls:
    - secretName: {{ .Values.tls.secretName }}


# Ingress.loki-stack.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  name: loki-stack
  namespace: loki-stack
spec:
  rules:
    - host: {{ .Value.host }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: loki-stack
                port:
                  number: 3100
  tls:
    - secretName: {{ .Values.tls.secretName }}

See Ingress | Kubernetes - https://kubernetes.io/docs/concepts/services-networking/ingress/ to leanr more.

Then, run kubectl apply command.

$ kubectl apply -f Ingress.loki-stack.yaml Ingress.loki-stack-grafana.yaml

$ kubectl get ingress -n loki-stack
NAME                 CLASS    HOSTS                                          ADDRESS        PORTS     AGE
loki-stack           <none>   loki-stack-k8s-s190.examples.cloudolife.com           10.233.43.59   80, 443   79m
loki-stack-grafana   <none>   loki-stack-grafana-k8s-s190.examples.cloudolife.com   10.233.43.59   80, 443   79m

Now, you can visite Grafana Loki with https://{{ .Values.host }}.