ImagePullBackOff Troubleshooting

Introduction

Sometimes we receive a ticket reporting that a pod is stuck in ImagePullBackOff or has an image pull issue. In this document, we cover common error types and how to troubleshoot them.

---

502 Bad Gateway

If you encounter a 502 Bad Gateway while pulling an image, first try to reproduce the issue.

Reproduce

1. SSH into the helper node of the cluster.
2. Try pulling the image manually.
3. Use the same image pull credentials as the affected pod.

In most cases the secret is hamdocker-credential, but always verify it in the pod spec.

• If you encounter the same error on the helper node, the problem is likely related to Hamstore.
• If the image pulls successfully from the helper node, check the node where the pod is scheduled:
  • Try pulling the same image from that node as well.
  • The issue might be network-related or caused by local node conditions (for example, not enough disk space to cache the image).

---

Debug Hamstore

If you reproduce the same issue on the helper node, start by checking Hamstore logs:

cd ~/hamstore
docker-compose logs -f --tail 100 | grep -i error

Review the logs and troubleshoot based on the error type.

Common Hamstore Errors

DNS issues

If you see an error like cannot resolve, check Hamstore DNS configuration:

1. Open the docker-compose.yml file under:

   ~/hamstore

2. Check the DNS configuration for the mitm and nginx containers.

3. Verify the configured resolvers and fix/replace them if needed.

Then restart the containers:

docker-compose up -d --force-recreate

Finally, try pulling the image again and confirm whether the issue is resolved.