Trying to access an Azure Container Registry with Private Link from home Mac over Azure Virtual Network Gateway always gives 403/access denied

Paul Field 21 Reputation points
2021-11-18T13:51:53.573+00:00

Hi. I am having this exact same error and have been pulling my hair out trying to solve it. It might be the underlying cause is different. I am not sure. I’ve been through the link above which gets you to validate your DNS setup, but it doesn't seem (unless I am misunderstanding, which is definitely possible) to give guidance on what to do if things still aren't working. I think it’s meant to say the DNS resolves to an external IP of the ACR if being accessed from outside. Or in internal IP if being accessed from a local vnet.

My objective is to connect from my home machine to a private container registry using Private Link (and so all public access on the ACR is turned off) which seems to be the recommended way. I then want to be able to push my containers into the registry with everything private.

(I've referred to my acr name as myacrname>

To reproduce the error:

And I use this to test every setting, I am doing:

az login
followed by
az acr login —name myacrname

It then says;

   Unable to get AAD authorization tokens with message: An error occurred: CONNECTIVITY_REFRESH_TOKEN_ERROR  
   Access to registry ‘myacrname.azurecr.io' was denied. Response code: 403. Please try running 'az login' again to refresh permissions.  
   Unable to get admin user credentials with message: Admin user is disabled.  

And it prompts me for a user name and password.

Using my Owner credentials, I always get;

   Error response from daemon: Get “myacrname://my .azurecr.io/v2/": denied: client with IP ‘myexternal' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.  
   Login failed.  

And where I’ve tried other (Service Principal or user setup under ACR/Networking/Access keys, with admin enabled) credentials, I’ve done below:

az acr login myacrname -u myacrname -p <password provided on that page>

But I always get the error

The login server endpoint suffix .azurecr.io is automatically omitted.

   Error response from daemon: Get "https://myacrname.azurecr.io/v2/": denied: client with IP ‘myexternalIP is not allowed access. Refer https://aka.ms/acr/firewall to grant access.  
   Login failed.  

As I understand it, the az login credentials expire in about an hour, so I am making sure I always do the az login for a test, just to be sure.

I’ve also tried connecting from VS code but get the same issue, ie it won’t connect and gives a 403 forbidden error.

For all these scenarios, if I look in the ACR logs, I always see the forbidden error but I am not sure if / how I can dig down deeper this way. One thing I did wonder is it mentions the User Agent as

My ACR setup so far:

I have an Azure Container Registry. Under Networking, Public Access is disabled and under Private Access a private end point is setup. It setup an end point for the login server of myacrname.azurecr.io and a data end point of myacrname.uksouth.data.azurecr.io

The private end point has a private DNS zone (setup by Azure)

Under customer visible FQDN I have:

myprivateendpointname.nic.<id> with two 10. IP addresses. One of them maps to the data end point and the other to the login end point.

There is one Configuration which is called privatelink-azurecr.io for private dns zone privatelink.azurecr.io which contains two FQDNs which map to the same two 10. IP addresses mention above.

I’ve setup a VPN from my end to a Virtual Network Gateway on the Azure side:

I’ve been following the VPN Point to Site instructions at https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal and at https://learn.microsoft.com/en-us/azure/vpn-gateway/point-to-site-about .

I’ve tried to do what it says at https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link for setup of ACR.

I am thinking that maybe the whole P2S isn’t the best way for dev / continuous integration but everything I am reading (and hopefully understanding but maybe not) is saying it could work, so I am just trying to see if it does.

I’ve created an IKEv2 VPN (from my Mac) with certificates and this connects fine, allocating me a 10. IP from the vnet I’ve setup. I followed the virtual network gateway instructions to do this.

I’ve tried all of the below with the VPN connected and not connected but it doesn’t make a difference.

I think is is because it’s more likely a connectivity issue.

As usual I am guessing it is probably something basic I am not seeing yet. I am pretty new to Azure so it’s definitely possible.

DNS settings:

Right now., I suspect the problem is DNS related but I can’t quite manage a solution yet.

If I do a dig for myacrname.azurecr.io from my machine, I get the external IP of the private link end point. And that’s with or without VPN.

If I dig the ACR from a Linux VM in the vnet, I get the 10. address of the ACR. And I can login with the method I mention below. I.e. az login followed by az acr login.

This may all be where my understanding of DNS is failing but my current theory (!) is that when I am connected to the virtual network gate way, my IP should appear to Azure as a 10. but it’s seeing the NAT address given out by my ISP. Does this make sense?

I looked at the DNS setup recommendations under https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns#virtual-network-and-on-premises-workloads-using-a-dns-forwarder and tried setting up an Azure Firewall / Policy / DNS settings to: Enable DNS settings to apply to the policy, use the Azure provided DNS server and DNS proxy enabled. Under that setting, it says;

To ensure DNS traffic is directed to the Azure Firewalls associated with this policy, you must configure your virtual network DNS server settings and set the Azure Firewall’s private IP address as a Custom DNS server.

I’ve done what it says which is, under vnet / DNS servers, I’ve got a DNS server address of the firewall public IP.

Should I be looking at other methods of DNS setup, e.g. setting up a VM and doing forwarding? This is suggested as another way of working but not sure it makes sense. But I was just experimenting.

I have also tried setting up a DNS server of in the Mac VPN config of 168.63.129.16

Other things tried:

I’ve tried turning off local firewalls, though didn’t think that would make a difference. And it didn’t.

Just as a side experiment, but wasn’t really thinking it would work, I’ve tried the Azure CLI add network-rule to allow my IP but that doesn’t help.

I’ve tried setting up an NSG for the vnet and allowing traffic from that IP but as I understand it, NSGs are over-riden for private end points, so this was a bit of a desperate trial.

Any ideas (or solutions) greatly appreciated. Thanks, Paul

Azure Container Registry
Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
387 questions
Azure VPN Gateway
Azure VPN Gateway
An Azure service that enables the connection of on-premises networks to Azure through site-to-site virtual private networks.
1,379 questions
Azure Firewall
Azure Firewall
An Azure network security service that is used to protect Azure Virtual Network resources.
565 questions
Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
461 questions
Remote Desktop
Remote Desktop
A Microsoft app that connects remotely to computers and to virtual apps and desktops.
4,240 questions
0 comments No comments
{count} votes

Accepted answer
  1. GitaraniSharma-MSFT 47,316 Reputation points Microsoft Employee
    2021-11-22T13:40:37.077+00:00

    Hello @Paul Field ,

    Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

    I discussed this issue with Azure Container engineer and below are our suggestions to fix the issue:

    Azure Networking:

    Yes, looks like DNS is the issue as you mentioned when you do a dig for myacrname.azurecr.io from your machine, you get the external IP of the private link end point. You should get the private IP instead.

    You followed the correct documentation but I would like to make sure that you are setting it up correctly:
    https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns#virtual-network-and-on-premises-workloads-using-a-dns-forwarder

    Make sure that the private endpoint DNS integration over Point to Site VPN connection is configured correctly:
    https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-P2S

    You have mentioned "I’ve done what it says which is, under vnet / DNS servers, I’ve got a DNS server address of the firewall public IP." - it should be Azure Firewall's private IP (not Public IP).

    NOTE : when you validate the solution, know that nslookup is not the right tool to make the private endpoint name resolution (DNS) validation over P2S VPN connections. Windows 10 has a feature called Network Resolution Policy Table (NRPT) and VPN connections will add DNS information inside NRPT. Because Nslookup is not aware of NPTR, you must use PowerShell Resolve-DNSName cmdlet to check DNS name resolution over P2S VPN connections

    If you want a more detailed understanding of Private endpoint DNS integration scenarios, please refer:
    https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-Scenarios

    Azure Container:

    Error 1 : Unable to get admin user credentials with message: Admin user is disabled.

    Solution : Enable Admin user
    https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli#authentication-options
    https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli#admin-account

    Error 2 : Get “myacrname://my .azurecr.io/v2/": denied: client with IP ‘myexternal' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.

    Cause : If your registry is configured for a virtual network with Private Link, IP network rules don't apply to the registry's private endpoints.

    Solution : https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link
    But I believe you have already followed this documentation and have checked all the settings and configurations.

    Additional troubleshooting document for your reference:
    https://learn.microsoft.com/en-us/azure/container-registry/container-registry-troubleshoot-access

    Kindly let us know if the above helps or you need further assistance on this issue.

    ----------------------------------------------------------------------------------------------------------------

    Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


1 additional answer

Sort by: Most helpful
  1. Ernest Richard XXLKPS5 1 Reputation point
    2021-12-04T16:20:55.073+00:00

    strong text

    0 comments No comments