question

PaulField-6375 avatar image
0 Votes"
PaulField-6375 asked ErnestRichardXXLKPS5-2115 answered

Trying to access an Azure Container Registry with Private Link from home Mac over Azure Virtual Network Gateway always gives 403/access denied

Hi. I am having this exact same error and have been pulling my hair out trying to solve it. It might be the underlying cause is different. I am not sure. I’ve been through the link above which gets you to validate your DNS setup, but it doesn't seem (unless I am misunderstanding, which is definitely possible) to give guidance on what to do if things still aren't working. I think it’s meant to say the DNS resolves to an external IP of the ACR if being accessed from outside. Or in internal IP if being accessed from a local vnet.

My objective is to connect from my home machine to a private container registry using Private Link (and so all public access on the ACR is turned off) which seems to be the recommended way. I then want to be able to push my containers into the registry with everything private.

(I've referred to my acr name as myacrname>

To reproduce the error:

And I use this to test every setting, I am doing:

az login
followed by
az acr login —name myacrname

It then says;

Unable to get AAD authorization tokens with message: An error occurred: CONNECTIVITY_REFRESH_TOKEN_ERROR
Access to registry ‘myacrname.azurecr.io' was denied. Response code: 403. Please try running 'az login' again to refresh permissions.
Unable to get admin user credentials with message: Admin user is disabled.

And it prompts me for a user name and password.

Using my Owner credentials, I always get;

Error response from daemon: Get “myacrname://my .azurecr.io/v2/": denied: client with IP ‘myexternal' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.
Login failed.


And where I’ve tried other (Service Principal or user setup under ACR/Networking/Access keys, with admin enabled) credentials, I’ve done below:

az acr login myacrname -u myacrname -p <password provided on that page>

But I always get the error

The login server endpoint suffix .azurecr.io is automatically omitted.
```
Error response from daemon: Get "https://myacrname.azurecr.io/v2/": denied: client with IP ‘myexternalIP is not allowed access. Refer https://aka.ms/acr/firewall to grant access.
Login failed.
```

As I understand it, the az login credentials expire in about an hour, so I am making sure I always do the az login for a test, just to be sure.

I’ve also tried connecting from VS code but get the same issue, ie it won’t connect and gives a 403 forbidden error.

For all these scenarios, if I look in the ACR logs, I always see the forbidden error but I am not sure if / how I can dig down deeper this way. One thing I did wonder is it mentions the User Agent as

My ACR setup so far:

I have an Azure Container Registry. Under Networking, Public Access is disabled and under Private Access a private end point is setup. It setup an end point for the login server of myacrname.azurecr.io and a data end point of myacrname.uksouth.data.azurecr.io

The private end point has a private DNS zone (setup by Azure)

Under customer visible FQDN I have:

myprivateendpointname.nic.<id> with two 10. IP addresses. One of them maps to the data end point and the other to the login end point.

There is one Configuration which is called privatelink-azurecr.io for private dns zone privatelink.azurecr.io which contains two FQDNs which map to the same two 10. IP addresses mention above.

I’ve setup a VPN from my end to a Virtual Network Gateway on the Azure side:

I’ve been following the VPN Point to Site instructions at https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-resource-manager-portal and at https://docs.microsoft.com/en-us/azure/vpn-gateway/point-to-site-about .

I’ve tried to do what it says at https://docs.microsoft.com/en-us/azure/container-registry/container-registry-private-link for setup of ACR.

I am thinking that maybe the whole P2S isn’t the best way for dev / continuous integration but everything I am reading (and hopefully understanding but maybe not) is saying it could work, so I am just trying to see if it does.

I’ve created an IKEv2 VPN (from my Mac) with certificates and this connects fine, allocating me a 10. IP from the vnet I’ve setup. I followed the virtual network gateway instructions to do this.

I’ve tried all of the below with the VPN connected and not connected but it doesn’t make a difference.

I think is is because it’s more likely a connectivity issue.

As usual I am guessing it is probably something basic I am not seeing yet. I am pretty new to Azure so it’s definitely possible.

DNS settings:

Right now., I suspect the problem is DNS related but I can’t quite manage a solution yet.

If I do a dig for myacrname.azurecr.io from my machine, I get the external IP of the private link end point. And that’s with or without VPN.

If I dig the ACR from a Linux VM in the vnet, I get the 10. address of the ACR. And I can login with the method I mention below. I.e. az login followed by az acr login.

This may all be where my understanding of DNS is failing but my current theory (!) is that when I am connected to the virtual network gate way, my IP should appear to Azure as a 10. but it’s seeing the NAT address given out by my ISP. Does this make sense?

I looked at the DNS setup recommendations under https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns#virtual-network-and-on-premises-workloads-using-a-dns-forwarder and tried setting up an Azure Firewall / Policy / DNS settings to: Enable DNS settings to apply to the policy, use the Azure provided DNS server and DNS proxy enabled. Under that setting, it says;

To ensure DNS traffic is directed to the Azure Firewalls associated with this policy, you must configure your virtual network DNS server settings and set the Azure Firewall’s private IP address as a Custom DNS server.

I’ve done what it says which is, under vnet / DNS servers, I’ve got a DNS server address of the firewall public IP.

Should I be looking at other methods of DNS setup, e.g. setting up a VM and doing forwarding? This is suggested as another way of working but not sure it makes sense. But I was just experimenting.

I have also tried setting up a DNS server of in the Mac VPN config of 168.63.129.16

Other things tried:

I’ve tried turning off local firewalls, though didn’t think that would make a difference. And it didn’t.

Just as a side experiment, but wasn’t really thinking it would work, I’ve tried the Azure CLI add network-rule to allow my IP but that doesn’t help.

I’ve tried setting up an NSG for the vnet and allowing traffic from that IP but as I understand it, NSGs are over-riden for private end points, so this was a bit of a desperate trial.

Any ideas (or solutions) greatly appreciated. Thanks, Paul

remote-desktop-clientazure-vpn-gatewayazure-firewallazure-private-linkazure-container-registry
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GitaraniSharmaMSFT-4262 avatar image
0 Votes"
GitaraniSharmaMSFT-4262 answered GitaraniSharmaMSFT-4262 commented

Hello @PaulField-6375 ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I discussed this issue with Azure Container engineer and below are our suggestions to fix the issue:

Azure Networking:

Yes, looks like DNS is the issue as you mentioned when you do a dig for myacrname.azurecr.io from your machine, you get the external IP of the private link end point. You should get the private IP instead.

You followed the correct documentation but I would like to make sure that you are setting it up correctly:
https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns#virtual-network-and-on-premises-workloads-using-a-dns-forwarder

Make sure that the private endpoint DNS integration over Point to Site VPN connection is configured correctly:
https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-P2S

You have mentioned "I’ve done what it says which is, under vnet / DNS servers, I’ve got a DNS server address of the firewall public IP." - it should be Azure Firewall's private IP (not Public IP).

NOTE : when you validate the solution, know that nslookup is not the right tool to make the private endpoint name resolution (DNS) validation over P2S VPN connections. Windows 10 has a feature called Network Resolution Policy Table (NRPT) and VPN connections will add DNS information inside NRPT. Because Nslookup is not aware of NPTR, you must use PowerShell Resolve-DNSName cmdlet to check DNS name resolution over P2S VPN connections

If you want a more detailed understanding of Private endpoint DNS integration scenarios, please refer:
https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-Scenarios

Azure Container:

Error 1 : Unable to get admin user credentials with message: Admin user is disabled.

Solution : Enable Admin user
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli#authentication-options
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli#admin-account

Error 2 : Get “myacrname://my .azurecr.io/v2/": denied: client with IP ‘myexternal' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.

Cause : If your registry is configured for a virtual network with Private Link, IP network rules don't apply to the registry's private endpoints.

Solution : https://docs.microsoft.com/en-us/azure/container-registry/container-registry-private-link
But I believe you have already followed this documentation and have checked all the settings and configurations.

Additional troubleshooting document for your reference:
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-troubleshoot-access

Kindly let us know if the above helps or you need further assistance on this issue.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi. Thanks for the very detailed answer. I've tried to setup the P2S solution from https://github.com/dmauser/PrivateLink/tree/master/DNS-Integration-P2S#solution-1-specify-dns-server-for-the-virtual-network but still have the same problem-- I.e. dig the acr from my mac and I get ext IP. Irrespective of vpn on or off. Also can't seem to get Resolve-DnsName to work on the mac as it doesn't seem to be present in pwsh and can't import it. Can you see anything wrong with what I've done?

Target solution:
151856-targetsolution.png
Firewall policy:
151798-fwpolicy.png
Vnet dns:
151807-vnetdns.png
Powershell issue:
151808-resolvedns.png


0 Votes 0 ·
targetsolution.png (336.0 KiB)

Hello @PaulField-6375 ,

Thank you for the update. I believe some of the screenshots did not come through like Firewall policy, Vnet DNS and PowerShell issue etc.

Coming back to the issue, I suspect the DNS query is not being forwarded to Azure via P2S VPN.

Let us try a different approach and see how that works. Let's remove the Azure Firewall and the custom DNS settings from the picture and try host files for DNS resolution. You can use the host file on your machine to override the DNS. Add the private IP address with the FQDN of the ACR in your hosts file and try again.

Refer the below blog on how to edit hosts file on Mac:
https://setapp.com/how-to/edit-mac-hosts-file

NOTE: The hosts file only overrides name resolution for the local computer. If you want to use a hosts file with multiple computers, you must modify it individually on each computer.

Regards,
Gita

0 Votes 0 ·
ErnestRichardXXLKPS5-2115 avatar image
0 Votes"
ErnestRichardXXLKPS5-2115 answered

strong text

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.