[Kubernetes (K8S) Ingress Nginx] Use Prometheus and Grafana for scraping the metrics of the Nginx Ingress controller

Posted on 2021-02-27 Edited on 2023-09-02 In Cloud Native , Kubernetes (K8S) , Ingress Nginx Views: Word count in article: 4.3k Reading time ≈ 4 mins.

Use Prometheus and Grafana for scraping the metrics of the Nginx Ingress controller

This tutorial will show you how to use Prometheus and Grafana for scraping the metrics of the Nginx Ingress controller.

Prerequisites

Kubernetes (K8S)

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

For more information about installing and using Kubernetes (K8s), see the Kubernetes (K8s) Docs.
Helm

Helm is the best way to find, share, and use software built for Kubernetes.
1
2
# Mac OS X
$ brew install helm
For more information about installing and using Helm, see the Helm Docs.
Nginx Ingress Controller

ingress-nginx is an Ingress controller for Kubernetes using Nginx as a reverse proxy and load balancer.

See [Installation Guide - Nginx Ingress Controller - https://kubernetes.github.io/ingress-nginx/deploy/] to learn more.
Prometheus

The Prometheus monitoring system includes a rich, multidimensional data model, a concise and powerful query language called PromQL, an efficient embedded timeseries database, and over 150 integrations with third-party systems.

See Installation | Grafana Labs - https://grafana.com/docs/loki/latest/installation/ to learn how to install Loki.

Or See Prometheus | Grafana Labs - https://grafana.com/oss/prometheus/ to learn how to install Prometheus.

Configuration

The controller should be configured for exporting metrics. This requires 3 configurations to the controller. These configurations are :

controller.metrics.enabled=true
controller.podAnnotations."prometheus.io/scrape"="true"
controller.podAnnotations."prometheus.io/port"="10254"

Helm

The easiest way to configure the controller for metrics is via helm upgrade. Assuming you have installed the ingress-nginx controller as a helm release named ingress-controller, then you can simply type the command show below :

$ helm upgrade ingress-controller ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set-string controller.podAnnotations."prometheus\.io/scrape"="true" \
--set-string controller.podAnnotations."prometheus\.io/port"="10254"

Or set values to values.yaml shown below;

# values.yaml

controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"

1
2
3

$ helm upgrade ingress-controller ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--values values.yaml

Manifest

If you are not using helm, you will have to edit your manifests like this:

Service manifest:

# manifests/Service.yaml

apiVersion: v1
kind: Service
metadata:
 annotations:
   prometheus.io/scrape: "true"
   prometheus.io/port: "10254"

spec:
  ports:
    - name: prometheus
      port: 10254
      targetPort: prometheus

Or DaemonSet manifest:

# manifests/DaemonSet.yaml

          ports:
            - name: prometheus
              containerPort: 10254

Run kubectl to apply manifests file

1	$ kubectl apply -f manifests/*.yaml

Grafana Dashboard

After the login you can import the Grafana dashboard from official dashboards, by following steps given below :

Left menu (hover over +) -> Dashboard
Click Import
Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json
Click Import JSON
Select the Prometheus data source
Click “Import”

ingress-nginx grafana

Then, follow the steps above to enter the copy pasted json from https://github.com/kubernetes/ingress-nginx/blob/main/deploy/grafana/dashboards/request-handling-performance.json

See ingress-nginx/deploy/grafana/dashboards at main · kubernetes/ingress-nginx - https://github.com/kubernetes/ingress-nginx/tree/main/deploy/grafana/dashboards to learn more.