App Service in internal App Service Environment v3 pulls docker-image over public outbound IP of ASE

Harald Reinmueller 1

I have an internal ASEv3 provisioned into my VNet. An App Service deployed in the ASEv3 needs to pull the container-image from an ACR in the same VNet. The ACR has disabled all public network access and uses a private endpoint for communication. The App Service tries to pull the docker-image over the public outbound IP address of the ASE which results in the following error-message inside the "Deployment Center - Logs" of my App Service:

ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myregistryxxxyyy.azurecr.io/v2/my-app/manifests/2021.1.7-appinsightsx: denied: client with IP '20.xx.xxx.xx' is not allowed access. Refer https://aka.ms/acr/firewall to grant access."}

My current workaround is to allow this public IP inside the firewall settings of the ACR.

But how can I tell the App Service to communicate over my VNet only? I already set the env-variables WEBSITE_VNET_ROUTE_ALL=1 and WEBSITE_PULL_IMAGE_OVER_VNET=true.

2 answers

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-08-04T14:57:36.687+00:00
Hi @Harald Reinmueller ,

When using regional VNET routing, make sure you have the following app settings configured

DOCKER_REGISTRY_SERVER_URL

DOCKER_REGISTRY_SERVER_USERNAME

DOCKER_REGISTRY_SERVER_PASSWORD

Otherwise, it will fall back to public route.

EDIT: See https://azure.github.io/AppService/2021/07/03/Linux-container-from-ACR-with-private-endpoint.html. You need to use Azure DNS so that it properly resolves the private endpoint within the VNET.
Please sign in to rate this answer.
Harald Reinmueller 1 Reputation point

2021-08-05T06:28:44.403+00:00

Hi @Ryan Hill ,

Thanks for your answer. My App Service uses system-assigned managed identity to authenticate against Azure Container Registry (ACR) and perform a docker pull operation. So I have no admin username/password configured.

I'm using this command to enable acrUseManagedIdentityCreds.

az resource update --ids $(az webapp show -g my-group -n my-app --query id --output tsv)/config/web \ --set properties.acrUseManagedIdentityCreds=True

How can I use acrUseManagedIdentityCreds=true and internal VNet routing?

Thanks, Harald

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-08-05T17:35:13.47+00:00

Hi @Harald Reinmueller ,

Have you already walked through configuring public IP access to your registry?

Harald Reinmueller 1 Reputation point

2021-08-06T10:07:18.503+00:00

Hi @Ryan Hill

yes that is my current workaround, to allow access to the public outbound IP of the ASE inside the firewall settings of the ACR.
But thats just a workaround. I want ASE + App Service to go VNet internal only.

Thanks, Harald

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-08-10T00:25:05.907+00:00

See my updated answer @Harald Reinmueller . Use Azure DNS to resolve the private endpoint.

Harald Reinmueller 1 Reputation point

2021-08-25T07:27:21.887+00:00

Hi @Ryan Hill

I retried with a private DNS zone in place and the container-registry admin username/password enabled like it is described inside your linked article.
The result is the same error-message (see above). The main difference to the infrastructure in the article is that I'm using an internal App Service Environment (v3) for hosting my App Services. And this App Service Environment has an public outbound IP address for communicating with public internet. And this IP address appears in the error-message of the container-registry.

So for me this is not an DNS issue, it's more a routing issue. Because the App Service Environment should not use its public outbound IP for communication with the container-registry in the same VNet. It should communicate with internal IP.

Thanks, Harald

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-09-02T23:01:08.707+00:00

Hi @Harald Reinmueller ,

I've spent some time verifying this and can confirm, whether your ASE is internal or external, that you will need to set DOCKER_REGISTRY_SERVER_USERNAME and DOCKER_REGISTRY_SERVER_PASSWORD in conjunction with WEBSITE_PULL_IMAGE_OVER_VNET=true to pull the images from ACR over private endpoint. You will also need to have a privatelink.azurecr.io link setup.

Verify your values in app settings and deployment center and restart the app. It should work.

Suresh Bettadapur 71 Reputation points

2021-09-27T12:39:35.33+00:00

I am also facing same issue .... all these are configured in my app service......

I have to enable outbound public IP of ASE in ACR ..... otherwise, requests fail

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-09-27T16:57:07.92+00:00

@Suresh Bettadapur can you provide a log snippet from the Deployment Center blade?

Suresh Bettadapur 71 Reputation points

2021-09-28T08:53:27.237+00:00

Getting error as per attachment -- expects outbound Public IP of ASE to be allowed in ACR

Ryan Hill 26,146 Reputation points Microsoft Employee

2021-09-28T18:25:43.047+00:00

Okay, and DOCKER_REGISTRY_SERVER_PASSWORD is configured to the access key of the registry?

Suresh Bettadapur 71 Reputation points

2021-09-29T06:34:21.643+00:00

Yes.

In ACR, Admin user access is enabled

In the app service Configuration, Docker Registry Server URL, User Name and Password are stored
Sign in to comment
Suresh Bettadapur 71 Reputation points

2022-06-08T07:35:15.667+00:00

Any other further update on this issue please? even now, the Pull requests to ACR from function app goes with oubtound public IP of ASE
Please sign in to rate this answer.

0 comments No comments
Sign in to comment