Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.
Depending on your setup you can route the AVD service traffic directly to the Azure network.
We recommend something like Azure Firewall to assist with this.
However, you can even perform this via UDR rules with ServiceTag support that is currently in public preview.
Please refer below documents:
https://learn.microsoft.com/en-us/azure/firewall/protect-windows-virtual-desktop
https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview
Hope this helps.
Please 'Accept as answer' if the provided information is helpful, so that it can help others in the community looking for help on similar topics.