Kubernetes (in AKS) error while pulling image from private JFrog artifactory pro: unable to fetch descriptor which reports content size of zero: invalid argument (or unexpected commit digest)

Question

The configuration is as follows,

AKS Cluster
Kubernetes tested Version: 1.22.2 | 1.21.2 | 1.20.9 (all listed have been tested)
Node OS-IMAGE: Ubuntu 18.04.6 LTS
Node KERNEL-VERSION: 5.4.0-1062-azure
Node CONTAINER-RUNTIME: containerd://1.4.9+azure | containerd://1.5.5+azure (all listed have been tested)

JFrog artifactory pro Version: 7.27.3

After created:

kubectl create secret docker-registry artifactory-pro -n test --docker-server=https://artifactory-pro. --docker-username= --docker-password=
or
kubectl create secret docker-registry artifactory-pro -n test --docker-server=artifactory-pro. --docker-username= --docker-password=

and put in deploymnent.yaml:

namespace: test
<...>
imagePullSecrets:

name: artifactory-pro

and run: kubectl apply -f deploymnent.yaml
image pull fails.

Error on AKS 1.21.2 and AKS 1.20.9 (containerd://1.4.9+azure):

Warning Failed 2m37s kubelet Error: ImagePullBackOff
Normal Pulling 2m24s (x2 over 2m40s) kubelet Pulling image "artifactory-pro.//:"
Warning Failed 2m23s (x2 over 2m37s) kubelet Failed to pull image "artifactory-pro.//:": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "artifactory-pro.//:": failed commit on ref "manifest-sha256:acab7797<...>": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:acab7797.<..>: failed precondition
Warning Failed 2m23s (x2 over 2m37s) kubelet Error: ErrImagePull

Error on AKS 1.22.2 (containerd://1.5.5+azure):

Warning Failed 20s kubelet Error: ImagePullBackOff
Normal Pulling 4s (x2 over 21s) kubelet Pulling image "artifactory-pro.//:"
Warning Failed 4s (x2 over 20s) kubelet Failed to pull image "artifactory-pro.//:": rpc error: code = InvalidArgument desc = failed to pull and unpack image "artifactory-pro.//:": unable to fetch descriptor (sha256:acab7797<...>) which reports content size of zero: invalid argument
Warning Failed 4s (x2 over 20s) kubelet Error: ErrImagePull

I have not found network problems between jfrog and azure aks. Logging in to jfrog is successful.
Downloading the same image from this artifactory pro on the docker (20.10.9 with containerd 1.4.11) on regular VM (CentOS 8) works correctly.
Downloading the same image from this artifactory pro on k8s ( v1.16.15 with CONTAINER-RUNTIME: docker://18.6.2 or docker://19.3.1 ) existing on regular VM (CentOS 7) works correctly.

I am asking for help, because I have not found such a case so far.
Please let me know if further details are required.

Answer

I know more now:

This is not an artifactory problem. Response for pull image request localy is OK.
The problem shows up when request is from the internet, because beetwen AKS and artifactory is Azure proxy (in another DC) (gatway for artifactory).

Summary:

localy Content-Length for HTTP HEAD request (downloading docker image manifets) is OK (real non 0 size), but
for HTTP HEAD request (downloading docker image manifets) from Internet, where network traffic is through a Azure proxy, Content-Length is set to 0 and containerd returns an error when pull docker image.

The cause of the problem is wrong setting Content-Length to 0, by proxy located in another Azure Cloud.

Does anyone know why this header is seting wrong by Azure proxy?

Kubernetes (in AKS) error while pulling image from private JFrog artifactory pro: unable to fetch descriptor which reports content size of zero: invalid argument (or unexpected commit digest)

1 answer