question

Pawegowski-4885 avatar image
0 Votes"
Pawegowski-4885 asked Pawegowski-4885 commented

Kubernetes (in AKS) error while pulling image from private JFrog artifactory pro: unable to fetch descriptor which reports content size of zero: invalid argument (or unexpected commit digest)

The configuration is as follows,

AKS Cluster
Kubernetes tested Version: 1.22.2 | 1.21.2 | 1.20.9 (all listed have been tested)
Node OS-IMAGE: Ubuntu 18.04.6 LTS
Node KERNEL-VERSION: 5.4.0-1062-azure
Node CONTAINER-RUNTIME: containerd://1.4.9+azure | containerd://1.5.5+azure (all listed have been tested)

JFrog artifactory pro Version: 7.27.3


After created:

kubectl create secret docker-registry artifactory-pro -n test --docker-server=https://artifactory-pro.<domain> --docker-username=<user> --docker-password=<pass>
or
kubectl create secret docker-registry artifactory-pro -n test --docker-server=artifactory-pro.<domain> --docker-username=<user> --docker-password=<pass>


and put in deploymnent.yaml:

namespace: test
<...>
imagePullSecrets:
- name: artifactory-pro


and run: kubectl apply -f deploymnent.yaml
image pull fails.


Error on AKS 1.21.2 and AKS 1.20.9 (containerd://1.4.9+azure):


Warning Failed 2m37s kubelet Error: ImagePullBackOff
Normal Pulling 2m24s (x2 over 2m40s) kubelet Pulling image "artifactory-pro.<domain>/<path>/<image>:<tag>"
Warning Failed 2m23s (x2 over 2m37s) kubelet Failed to pull image "artifactory-pro.<domain>/<path>/<image>:<tag>": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "artifactory-pro.<domain>/<path>/<image>:<tag>": failed commit on ref "manifest-sha256:acab7797<...>": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:acab7797.<..>: failed precondition
Warning Failed 2m23s (x2 over 2m37s) kubelet Error: ErrImagePull

Error on AKS 1.22.2 (containerd://1.5.5+azure):


Warning Failed 20s kubelet Error: ImagePullBackOff
Normal Pulling 4s (x2 over 21s) kubelet Pulling image "artifactory-pro.<domain>/<path>/<image>:<tag>"
Warning Failed 4s (x2 over 20s) kubelet Failed to pull image "artifactory-pro.<domain>/<path>/<image>:<tag>": rpc error: code = InvalidArgument desc = failed to pull and unpack image "artifactory-pro.<domain>/<path>/<image>:<tag>": unable to fetch descriptor (sha256:acab7797<...>) which reports content size of zero: invalid argument
Warning Failed 4s (x2 over 20s) kubelet Error: ErrImagePull


I have not found network problems between jfrog and azure aks. Logging in to jfrog is successful.
Downloading the same image from this artifactory pro on the docker (20.10.9 with containerd 1.4.11) on regular VM (CentOS 8) works correctly.
Downloading the same image from this artifactory pro on k8s ( v1.16.15 with CONTAINER-RUNTIME: docker://18.6.2 or docker://19.3.1 ) existing on regular VM (CentOS 7) works correctly.

I am asking for help, because I have not found such a case so far.
Please let me know if further details are required.

azure-kubernetes-serviceazure-container-registry
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Pawegowski-4885 , thank you for your question.

Although I could not reproduce the issue, there are many moving parts in terms of how the registry and the image itself is set up. My best guess would be a containerd issue somewhere similar to this.

The best course of action would be to open a Microsoft technical Support Request. Also, once you get the issue fixed, request you to reply back here on the thread with the resolution steps for the benefit of the community.

0 Votes 0 ·

I know more now:

This is not an artifactory problem. Response for pull image request localy is OK.
The problem shows up when request is from the internet, because beetwen AKS and artifactory is Azure proxy (in another DC) (gatway for artifactory).

Summary:

localy Content-Length for HTTP HEAD request (downloading docker image manifets) is OK (real non 0 size), but
for HTTP HEAD request (downloading docker image manifets) from Internet, where network traffic is through a Azure proxy, Content-Length is set to 0 and containerd returns an error when pull docker image.


The cause of the problem is wrong setting Content-Length to 0, by proxy located in another Azure Cloud.

Does anyone know why this header is seting wrong by Azure proxy?

0 Votes 0 ·

1 Answer

Pawegowski-4885 avatar image
1 Vote"
Pawegowski-4885 answered

I know more now:

This is not an artifactory problem. Response for pull image request localy is OK.
The problem shows up when request is from the internet, because beetwen AKS and artifactory is Azure proxy (in another DC) (gatway for artifactory).

Summary:

localy Content-Length for HTTP HEAD request (downloading docker image manifets) is OK (real non 0 size), but
for HTTP HEAD request (downloading docker image manifets) from Internet, where network traffic is through a Azure proxy, Content-Length is set to 0 and containerd returns an error when pull docker image.


The cause of the problem is wrong setting Content-Length to 0, by proxy located in another Azure Cloud.

Does anyone know why this header is seting wrong by Azure proxy?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.