[Kubernetes (K8S) FAQs] Troubleshooting Kubernetes Pod Startup Issues: Image Pull Failures
Troubleshooting Kubernetes Pod Startup Issues: Image Pull Failures
Introduction
Recently, I encountered an issue where a Kubernetes (K8s) pod failed to start, getting stuck at:
1 | Pulling image "docker.io/bitnami/minideb:stretch" |
However, running docker images
on the node showed that the image was already present.
1 | docker images | grep bitnami/minideb |
Root Cause Analysis
Upon deeper investigation, the issue was caused by:
-
imagePullPolicy: Always
- Kubernetes was trying to pull the image every time, even though it was available locally.
- This setting forces K8s to always fetch the latest image, which isn’t ideal for stable images that don’t change frequently.
-
Disk Usage Exceeding 70%
- When disk usage goes beyond this threshold, Docker automatically starts cleaning up unused images.
- The required image was deleted, causing the pod to fail during the pull operation.
- Manually pulling the image using
docker pull
resolved this part of the issue.
-
Need to Inspect Logs Directly on the Node
- GUI tools like Lens had outdated cache and continued displaying unrelated image pull errors from previous failed attempts.
- Running
kubectl describe pod <pod-name>
directly on the K8s node provided the most accurate and real-time information.
Solution
To prevent such issues in the future, I applied the following fixes:
-
Modify Image Pull Policy
1
imagePullPolicy: IfNotPresent
This ensures K8s only pulls the image if it is not already present locally.
-
Monitor Disk Usage
- Regularly check disk usage with
df -h
and clean up unused Docker resources using:docker system prune -a
- Regularly check disk usage with
-
Use CLI Over GUI for Debugging
- Always verify pod issues directly on the node using:
kubectl describe pod <pod-name>
- This avoids misleading information caused by GUI caching issues.
- Always verify pod issues directly on the node using:
Conclusion
This troubleshooting process reinforced the importance of understanding K8s image pull policies, monitoring disk space, and relying on CLI tools for accurate debugging. Hopefully, these insights help others facing similar Kubernetes startup issues! 🚀
References
[4] Docker Disk Cleanup Guide - https://docs.docker.com/config/pruning/