[Infrastructure as Code (IaC) Pulumi] Use Pulumi kubernetes (K8S) Helm Chart to deploy Fluentd

Posted on 2021-11-20 Edited on 2023-09-02 In Infrastructure as Code (IaC) , Cloud Native , Kubernetes (K8S) , Pulumi Views: Word count in article: 4.7k Reading time ≈ 4 mins.

Fluentd

Fluentd is an open source data collector for unified logging layer.

Fluentd allows you to unify data collection and consumption for a better use and understanding of data.

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on. Fluentd helps you unify your logging infrastructure (Learn more about the Unified Logging Layer).

This article is about how to use Pulumi, kubernetes (K8S) provider, Helm Chart and TypeScript SDK to deploy Fluentd within Kubernetes (K8S).

Prerequisites

Kubernetes (K8S)

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

See Getting started | Kubernetes - https://kubernetes.io/docs/setup/ to leanr more.
Pulumi - Modern Infrastructure as Code - https://www.pulumi.com/

Pulumi is a modern infrastructure-as-code platform that allows you to use common programming languages, tools, and frameworks, to provision, update, and manage cloud infrastructure resources.

Install the Pulumi - https://www.pulumi.com/ CLI.
1
2
# Mac OS X
$ brew install pulumi
See Download and Install | Pulumi - https://www.pulumi.com/docs/get-started/install/ to learn more about others OS.
Node.js - https://nodejs.org/en/

Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine.

Install Node.js - https://nodejs.org/en/ CLI.
1
2
# Mac OS X
$ brew install node
See Node.js - https://nodejs.org/en/ to learn more about others OS.

Usage

Pulumi New

Create the workspace directory.

1
2
3

$ mkdir -p col-example-pulumi-typescript-fluentd

$ cd col-example-pulumi-typescript-fluentd

Pulumi login into local file system.

1
2
3

$ pulumi login file://.
Logged in to cloudolife as cloudolife (file://.)
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.

Pulumi new a project with kubernetes-typescript SDK.

1	$ pulumi new kubernetes-typescript

The above command will create some files within the current directory.

tree . -L 1
.
├── node_modules/
├── package.json
├── package.json.lock
├── Pulumi.dev.yaml
├── Pulumi.yaml
└── main.ts

Install js-yaml package to load and parse yaml file.

1	$ npm i js-yaml

Pulumi Configuration

Configure Kubernetes

By default, Pulumi will look for a kubeconfig file in the following locations, just like kubectl:

The environment variable: $KUBECONFIG,
Or in current user’s default kubeconfig directory: ~/.kube/config

If the kubeconfig file is not in either of these locations, Pulumi will not find it, and it will fail to authenticate against the cluster. Set one of these locations to a valid kubeconfig file, if you have not done so already.

Configure Values.yaml

Edit values.yaml and replace content within {{ }}.

# values.yaml

# helm-charts/values.yaml at main · fluent/helm-charts
# https://github.com/fluent/helm-charts/blob/main/charts/fluentd/values.yaml

# ## Additional environment variables to set for fluentd pods
# env:
#   - name: FLUENT_ELASTICSEARCH_HOST
#     value: "elasticsearch-master.elastic"

# ## Fluentd configurations:
# ##
# fileConfigs:    
#   04_outputs.conf: |-
#     <label @OUTPUT>
#       <match **>
#         @type elasticsearch
#         host "elasticsearch-master.elastic"
#         port 9200
#         path ""
#         user elastic
#         password changeme
#       </match>
#     </label>

main.ts

// main.ts

import * as pulumi from "@pulumi/pulumi";

import * as k8s from "@pulumi/kubernetes";

const yaml = require('js-yaml');
const fs   = require('fs');

const nameElastic = "elastic"

// kubernetes.core/v1.Namespace | Pulumi
// https://www.pulumi.com/docs/reference/pkg/kubernetes/core/v1/namespace/
const namespaceElastic = new k8s.core.v1.Namespace(nameElastic, {
    metadata: {
        name: nameElastic,
    },
})

const values = yaml.safeLoad(fs.readFileSync("./values.yaml", 'utf8'))

const charNameFluentd = "fluentd"

const charFluentd = new k8s.helm.v3.Chart(charNameFluentd, {
    chart: charNameFluentd,
    version: "0.2.12",
    fetchOpts:{
        repo: "https://fluent.github.io/helm-charts",
    },
    values: values,
    namespace: namespaceElastic.metadata.name,
})

Pulumi Up

Run pulumi up to create the namespace and pods.

1	$ pulumi up

See pods about fluentd.

$ kubectl get pods | grep fluentd
default                                     fluentd-pzkb8                                                1/1     Running            304 (26m ago)     7d15h
default                                     fluentd-wlc8g                                                1/1     Running            308 (15h ago)     7d15h
default                                     fluentd-xxf6t                                                1/1     Running            96 (18h ago)      7d15h