[Kubernetes (K8S) FAQs] Troubleshooting Kubernetes Pod Startup Issues: Image Pull Failures

Troubleshooting Kubernetes Pod Startup Issues: Image Pull Failures

Introduction

Recently, I encountered an issue where a Kubernetes (K8s) pod failed to start, getting stuck at:

1
Pulling image "docker.io/bitnami/minideb:stretch"

However, running docker images on the node showed that the image was already present.

1
2
docker images | grep bitnami/minideb
bitnami/minideb stretch e398a222dbd6 2 years ago 53.8MB

Root Cause Analysis

Upon deeper investigation, the issue was caused by:

  1. imagePullPolicy: Always

    • Kubernetes was trying to pull the image every time, even though it was available locally.
    • This setting forces K8s to always fetch the latest image, which isn’t ideal for stable images that don’t change frequently.
  2. Disk Usage Exceeding 70%

    • When disk usage goes beyond this threshold, Docker automatically starts cleaning up unused images.
    • The required image was deleted, causing the pod to fail during the pull operation.
    • Manually pulling the image using docker pull resolved this part of the issue.
  3. Need to Inspect Logs Directly on the Node

    • GUI tools like Lens had outdated cache and continued displaying unrelated image pull errors from previous failed attempts.
    • Running kubectl describe pod <pod-name> directly on the K8s node provided the most accurate and real-time information.

Solution

To prevent such issues in the future, I applied the following fixes:

  1. Modify Image Pull Policy

    1
    imagePullPolicy: IfNotPresent

    This ensures K8s only pulls the image if it is not already present locally.

  2. Monitor Disk Usage

    • Regularly check disk usage with df -h and clean up unused Docker resources using:
      docker system prune -a
      
  3. Use CLI Over GUI for Debugging

    • Always verify pod issues directly on the node using:
      kubectl describe pod <pod-name>
      
    • This avoids misleading information caused by GUI caching issues.

Conclusion

This troubleshooting process reinforced the importance of understanding K8s image pull policies, monitoring disk space, and relying on CLI tools for accurate debugging. Hopefully, these insights help others facing similar Kubernetes startup issues! 🚀

References

[1] Kubernetes Image Pull Policy - https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

[2] Troubleshooting Kubernetes Pods - https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/

[3] Kubernetes Node Disk Pressure and Eviction Policy - https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/

[4] Docker Disk Cleanup Guide - https://docs.docker.com/config/pruning/

[5] crictl Command Reference (For Kubernetes CRI Debugging) - https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/