[Ingress Nginx] Persisting Logs in Ingress Nginx

Persisting Logs in Ingress Nginx

When running an application at scale in a Kubernetes environment, logging is essential for monitoring, debugging, and auditing. Ingress Nginx, a popular ingress controller, plays a critical role in managing and routing traffic into your Kubernetes cluster. This article will walk you through how to persist logs in Ingress Nginx, including how to configure different types of logs such as controller logs, access logs, and error logs.

Introduction

The logging in NGINX Ingress Controller consists of three main parts:

1. Controller Logs: These are the logs for the NGINX Ingress Controller itself. By default, these logs are output to stdout. However, you can configure them to be output to a file by using the --log_dir parameter during startup. If redirected to a file, the logs will automatically rotate but will not be cleaned up automatically.

2. Access Logs: These logs record the incoming requests handled by NGINX. They are output to stdout by default but can be configured to output to a specific file through the NGINX configuration.

3. Error Logs: These logs capture any errors that occur during the processing of requests. Like access logs, error logs are output to stderr by default but can also be redirected to a specific file.

Writing Logs to Disk

To persist logs on the disk, follow these steps:

1. Create a Directory for Logs

   On the node where Ingress Nginx is running, create a directory for storing logs. Ensure that the directory has appropriate permissions.

   mkdir -pv /var/lib/docker/nginxlogs/ingress
   chown -r 755:755 /var/lib/docker/nginxlogs/ingress

2. Configure the Controller Logs

   You can configure the Ingress Nginx Controller to output logs to a file by setting the --log_dir parameter. Here’s an example of how you can modify the controller’s arguments:

   args:
     - /nginx-ingress-controller
     - --configmap=$(POD_NAMESPACE)/nginx-configuration
     - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
     - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
     - --publish-service=$(POD_NAMESPACE)/ingress-nginx
     - --annotations-prefix=nginx.ingress.kubernetes.io
     - --log_dir=/var/log/nginx/
     - --logtostderr=false

3. Modify ConfigMap for Access and Error Logs

   Update the ConfigMap to define where access logs and error logs should be stored. You can also specify the format of these logs.

   worker-processes: "4"
   use-forwarded-headers: "true"
   log-format-upstream: "[$host] [$remote_addr] [$http_x_forwarded_for] [$remote_user] [$time_local] [$request] [$status] [$body_bytes_sent] [$request_time] [$upstream_addr] [$upstream_response_time] [$connection] [$connection_requests] [$msec] [$uri] [$body_bytes_sent] [$http_referer] [$http_user_agent] [$request_length] [$http_session_id]"
   access-log-path: "/var/log/nginx/access.log"
   error-log-path: "/var/log/nginx/error.log"

4. Mount the Log Directory

   Finally, ensure that the logs are written to a persistent directory by mounting it within your pod configuration:

   volumeMounts:
     - mountPath: /etc/localtime
       name: localtime
       readOnly: true
     - mountPath: /var/log/nginx
       name: app-log
   volumes:
     - name: localtime
       hostPath:
         path: /etc/localtime
     - name: app-log
       hostPath:
         path: "/var/lib/docker/nginxlogs/ingress"

Example Output

After implementing the above configuration, the logs should appear in the specified directory on the host. For example:

[root@ingress]# ll
total 12
-rw-r--r-- 1 33 tape    0 March 23 09:25 access.log
-rw-r--r-- 1 33 tape    0 March 23 09:25 error.log
-rw-r--r-- 1 33 tape  265 March 23 09:25 nginx-ingress-controller.k8s-node-13.www-data.log.ERROR.20200423-092512.6
-rw-r--r-- 1 33 tape 2996 March 23 09:25 nginx-ingress-controller.k8s-node-13.www-data.log.INFO.20200423-092510.6
-rw-r--r-- 1 33 tape  543 March 23 09:25 nginx-ingress-controller.k8s-node-13.www-data.log.WARNING.20200423-092510.6
lrwxrwxrwx 1 33 tape   82 March 23 09:25 nginx-ingress-controller.ERROR -> nginx-ingress-controller.k8s-node-13.www-data.log.ERROR.20200323-092512.6
lrwxrwxrwx 1 33 tape   81 March 23 09:25 nginx-ingress-controller.INFO -> nginx-ingress-controller.k8s-node-13.www-data.log.INFO.20200323-092510.6
lrwxrwxrwx 1 33 tape   84 March 23 09:25 nginx-ingress-controller.WARNING -> nginx-ingress-controller.k8s-node-13.www-data.log.WARNING.20200323-092510.6

Domain-Specific Access Logs

If you wish to have logs that are specific to a domain, you can modify the Ingress resource directly. However, there is no global variable that allows automatic inclusion of the domain in logs. Here is how you can do it manually:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-test
  namespace: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      access_log /var/log/nginx/test.cloudolife.com.log;
spec:
  rules:
  - host: test.sy.com
    http:
      paths:
      - path: /
        backend:
          serviceName: tomcat-test
          servicePort: 6080

This configuration will create an access log specific to the test.cloudolife.com domain.

Conclusion

Persisting logs in Ingress Nginx involves configuring the controller, access logs, and error logs to write to specific files. This setup ensures that logs are not lost during pod restarts and can be retained for auditing and debugging purposes. While the default setup directs logs to stdout and stderr, configuring them to write to disk provides more control and persistence. If you’re dealing with multiple domains, consider manually setting up domain-specific access logs for better granularity.