[Kubernetes (K8S)] Helm install Grafana Loki to deploy a log aggregation system within Kubernetes (K8S)

Posted on Edited on In , , Views: Word count in article: 5.5k Reading time ≈ 5 mins.

Loki: like Prometheus, but for logs

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

  • does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

  • indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.

  • is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
    has native support in Grafana (needs Grafana v6.0).

A Loki-based logging stack consists of 3 components:

  • promtail is the agent, responsible for gathering logs and sending them to Loki.

  • loki is the main server, responsible for storing logs and processing queries.

  • Grafana for querying and displaying the logs.

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

This article is about how to use Helm to deploy Loki Stack (Loki, Promtail, Grafana, Prometheus) on Kubernetes (K8S).

Prerequisites

  • Kubernetes (K8S)
    Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

    For more information about installing and using Kubernetes (K8s), see the Kubernetes (K8s) Docs.

  • Helm
    Helm is the best way to find, share, and use software built for Kubernetes.

    For more information about installing and using Helm, see the Helm Docs.

Installation

Custom Values.yaml

Remember to replace nfs with your prefer storage class name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# values.yaml

# loki/values.yaml at master · grafana/loki
# https://github.com/grafana/loki/blob/master/production/helm/loki-stack/values.yaml

# Deploy Loki Stack (Loki, Promtail, Grafana, Prometheus) with persistent volume claim
# https://grafana.com/docs/loki/latest/installation/helm/#deploy-loki-stack-loki-promtail-grafana-prometheus-with-persistent-volume-claim

grafana:
  enabled: true
  
  # helm-charts/values.yaml at main · grafana/helm-charts
  # https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml

  ## Enable persistence using Persistent Volume Claims
  ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
  ##
  persistence:
    # type: pvc
    enabled: enable
    storageClassName: nfs
    # accessModes:
    #   - ReadWriteOnce
    # size: 10Gi
    # # annotations: {}
    # finalizers:
    #   - kubernetes.io/pvc-protection
    # selectorLabels: {}
    # subPath: ""
    # existingClaim:

prometheus:
  enabled: true

  alertmanager:
    persistentVolume:
      enabled: true
      
  server:
    persistentVolume:
      enabled: true
      
loki:
  persistence:
    enabled: true
    
    storageClassName: nfs
    size: 50Gi

Install by Helm

Helm install loki-stack into loki-stack namespace.

1
2
3
4
5
6
7
8
9
10
11
# crate namespace:
$ kubectl create namespace loki-stack

# Add the Helm repository:
$ helm repo add loki https://grafana.github.io/loki/charts

# Update your local Helm chart repository cache:
$ helm repo update

# To install Helm chart:
$ helm install loki-stack loki/loki-stack -n loki-stack -f values.yaml

See Helm release about loki-stack

1
2
3
$ helm list --namespace loki-stack
NAME      	NAMESPACE 	REVISION	UPDATED                               	STATUS  	CHART           	APP VERSION
loki-stack	loki-stack	1       	2020-11-07 16:01:20.573601 +0800 +0800	deployed	loki-stack-2.0.0	v2.0.0

See pods about loki-stack.

1
2
3
4
5
$ kubectl get pods -n loki-stack
NAME                                  READY   STATUS    RESTARTS   AGE
loki-stack-0                          0/1     Running   2095       10d
loki-stack-grafana-64d4d55957-5f7pv   1/1     Running   6          13d
loki-stack-promtail-5f9fr             1/1     Running   7          10d

Run Loki behind HTTPS ingress

If Loki and Promtail are deployed on different clusters you can add an Ingress in front of Loki. By adding a certificate you create an HTTPS endpoint. For extra security you can also enable Basic Authentication on the Ingress.

Promtail enable HTTPS

In Promtail, set the following values to communicate using HTTPS and basic authentication:

1
2
3
4
loki:
  serviceScheme: https
  user: user
  password: pass

Sample Helm template for Ingress:

An Ingress example for loki-stack.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Ingress.loki.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: {{ .Values.ingress.class }}
    ingress.kubernetes.io/auth-type: "basic"
    ingress.kubernetes.io/auth-secret: {{ .Values.ingress.basic.secret }}
  name: loki-stack
  namespace: loki-stack
spec:
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
      - backend:
          serviceName: loki-stack
          servicePort: 3100
  tls:
  - secretName: {{ .Values.ingress.cert }}
    hosts:
    - {{ .Values.ingress.host }}

Then kubectl apply it.

1
$ kubectl apply -f Ingress.loki.yaml

An Ingress example for loki-stack-grafana.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Ingress.loki-stack-grafana.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: {{ .Values.ingress.class }}
    kubernetes.io/tls-acme: {{ .Values.ingress.acme }}
  name: loki-stack-grafana
  namespace: loki-stack
spec:
  rules:
  - host: l{{ .Values.ingress.host }} 
    http:
      paths:
      - backend:
          serviceName: loki-stack-grafana
          servicePort: 80
  tls:
  - secretName: {{ .Values.ingress.cert }}
    hosts:
    - {{ .Values.ingress.host }}

Then kubectl apply it.

1
$ kubectl apply -f Ingress.loki.yaml

References

[1] Helm | Grafana Labs - https://grafana.com/docs/loki/latest/installation/helm/

[2] loki-stack 2.0.2 · helm/loki - https://artifacthub.io/packages/helm/loki/loki-stack

[3] grafana/loki: Like Prometheus, but for logs. - https://github.com/grafana/loki

[4] Helm - https://helm.sh/

[5] Kubernetes - https://kubernetes.io/