Networking considerations for an App Service Environment
Azure App Service Environment is a deployment of Azure App Service into a subnet in your Azure virtual network (VNet). There are two deployment types for an App Service environment (ASE):
- External ASE: Exposes the ASE-hosted apps on an internet-accessible IP address. For more information, see Create an External ASE.
- ILB ASE: Exposes the ASE-hosted apps on an IP address inside your VNet. The internal endpoint is an internal load balancer (ILB), which is why it's called an ILB ASE. For more information, see Create and use an ILB ASE.
There are two versions of App Service Environment: ASEv1 and ASEv2. For information on ASEv1, see Introduction to App Service Environment v1. ASEv1 can be deployed in a classic or Resource Manager VNet. ASEv2 can only be deployed into a Resource Manager VNet.
All calls from an ASE that go to the internet leave the VNet through a VIP assigned for the ASE. The public IP of this VIP is the source IP for all calls from the ASE that go to the internet. If the apps in your ASE make calls to resources in your VNet or across a VPN, the source IP is one of the IPs in the subnet used by your ASE. Because the ASE is within the VNet, it can also access resources within the VNet without any additional configuration. If the VNet is connected to your on-premises network, apps in your ASE also have access to resources there without additional configuration.
If you have an External ASE, the public VIP is also the endpoint that your ASE apps resolve to for:
- Web deployment.
- Remote debugging.
If you have an ILB ASE, the address of the ILB is the endpoint for HTTP/S, FTP/S, web deployment, and remote debugging.
The normal app access ports are:
|HTTP/HTTPS||User configurable||80, 443|
|FTP/FTPS||User configurable||21, 990, 10001-10020|
|Visual Studio remote debugging||User configurable||4020, 4022, 4024|
This is true if you're on an External ASE or on an ILB ASE. If you're on an External ASE, you hit those ports on the public VIP. If you're on an ILB ASE, you hit those ports on the ILB. If you lock down port 443, there can be an effect on some features exposed in the portal. For more information, see Portal dependencies.
ASE subnet size
The size of the subnet used to host an ASE cannot be altered after the ASE is deployed. The ASE uses an address for each infrastructure role as well as for each Isolated App Service plan instance. Additionally, there are 5 addresses used by Azure Networking for every subnet that is created. An ASE with no App Service plans at all will use 12 addresses before you create an app. If it is an ILB ASE then it will use 13 addresses before you create an app in that ASE. As you scale out your ASE, infrastructure roles are added every multiple of 15 and 20 of your App Service plan instances.
Nothing else can be in the subnet but the ASE. Be sure to choose an address space that allows for future growth. You can't change this setting later. We recommend a size of
/24 with 256 addresses.
When you scale up or down, new roles of the appropriate size are added and then your workloads are migrated from the current size to the target size. Only after your apps are migrated are the original VMs removed. This means that if you had an ASE with 100 ASP instances there would be a period where you need double the number of VMs. It is for this reason that we recommend the use of a '/24' to accommodate any changes you might require.
ASE inbound dependencies
The ASE inbound access dependencies are:
|Management||App Service management addresses||ASE subnet: 454, 455|
|ASE internal communication||ASE subnet: All ports||ASE subnet: All ports|
|Allow Azure load balancer inbound||Azure load balancer||ASE subnet: All ports|
|App assigned IP addresses||App assigned addresses||ASE subnet: All ports|
The inbound management traffic provides command and control of the ASE in addition to system monitoring. The source addresses for this traffic are listed in the ASE Management addresses document. The network security configuration needs to allow access from all IPs on ports 454 and 455. If you block access from those addresses, your ASE will become unhealthy and then become suspended.
Within the ASE subnet there are many ports used for internal component communication and they can change. This requires all of the ports in the ASE subnet to be accessible from the ASE subnet.
For the communication between the Azure load balancer and the ASE subnet the minimum ports that need to be open are 454, 455 and 16001. The 16001 port is used for keep alive traffic between the load balancer and the ASE. If you are using an ILB ASE then you can lock traffic down to just the 454, 455, 16001 ports. If you are using an External ASE then you need to take into account the normal app access ports. If you are using app assigned addresses you need to open it to all ports. When an address is assigned to a specific app, then the load balancer will use ports that are not known of in advance to send HTTP and HTTPS traffic to the ASE.
If you are using app assigned IP addresses you need to allow traffic from the IPs assigned to your apps to the ASE subnet.
The TCP traffic that comes in on ports 454 and 455 must go back out from the same VIP or you will have an asymmetric routing problem.
ASE outbound dependencies
For outbound access, an ASE depends on multiple external systems. Many of those system dependencies are defined with DNS names and don't map to a fixed set of IP addresses. Thus, the ASE requires outbound access from the ASE subnet to all external IPs across a variety of ports.
The complete list of outbound dependencies are listed in the document that describes Locking down App Service Environment outbound traffic. If the ASE loses access to its dependencies, it stops working. When that happens long enough, the ASE is suspended.
If the VNet is configured with a customer-defined DNS server, the tenant workloads use it. The ASE still needs to communicate with Azure DNS for management purposes.
If the VNet is configured with a customer DNS on the other side of a VPN, the DNS server must be reachable from the subnet that contains the ASE.
To test resolution from your web app you can use the console command nameresolver. Go to the debug window in your scm site for your app or go to the app in the portal and select console. From the shell prompt you can issue the command nameresolver along with the address you wish to look up. The result you get back is the same as what your app would get while making the same lookup. If you use nslookup you will do a lookup using Azure DNS instead.
If you change the DNS setting of the VNet that your ASE is in, you will need to reboot your ASE. To avoid rebooting your ASE, it is highly recommended that you configure your DNS settings for your VNet before you create your ASE.
In addition to the ASE functional dependencies, there are a few extra items related to the portal experience. Some of the capabilities in the Azure portal depend on direct access to SCM site. For every app in Azure App Service, there are two URLs. The first URL is to access your app. The second URL is to access the SCM site, which is also called the Kudu console. Features that use the SCM site include:
- Web jobs
- Log streaming
- Process Explorer
When you use an ILB ASE, the SCM site isn't internet accessible from outside the VNet. When your app is hosted on an ILB ASE, some capabilities will not work from the portal.
Many of those capabilities that depend upon the SCM site are also available directly in the Kudu console. You can connect to it directly rather than by using the portal. If your app is hosted in an ILB ASE, use your publishing credentials to sign in. The URL to access the SCM site of an app hosted in an ILB ASE has the following format:
<appname>.scm.<domain name the ILB ASE was created with>
If your ILB ASE is the domain name contoso.net and your app name is testapp, the app is reached at testapp.contoso.net. The SCM site that goes with it is reached at testapp.scm.contoso.net.
Functions and Web jobs
Both Functions and Web jobs depend on the SCM site but are supported for use in the portal, even if your apps are in an ILB ASE, so long as your browser can reach the SCM site. If you are using a self-signed certificate with your ILB ASE, you will need to enable your browser to trust that certificate. For IE and Edge that means the certificate has to be in the computer trust store. If you are using Chrome then that means you accepted the certificate in the browser earlier by presumably hitting the scm site directly. The best solution is to use a commercial certificate that is in the browser chain of trust.
ASE IP addresses
An ASE has a few IP addresses to be aware of. They are:
- Public inbound IP address: Used for app traffic in an External ASE, and management traffic in both an External ASE and an ILB ASE.
- Outbound public IP: Used as the "from" IP for outbound connections from the ASE that leave the VNet, which aren't routed down a VPN.
- ILB IP address: If you use an ILB ASE.
- App-assigned IP-based SSL addresses: Only possible with an External ASE and when IP-based SSL is configured.
All these IP addresses are easily visible in an ASEv2 in the Azure portal from the ASE UI. If you have an ILB ASE, the IP for the ILB is listed.
These IP addresses will not change so long as your ASE stays up and running. If your ASE becomes suspended and restored, the addresses used by your ASE will change. The normal cause for an ASE to become suspended is if you block inbound management access or block access to an ASE dependency.
App-assigned IP addresses
With an External ASE, you can assign IP addresses to individual apps. You can't do that with an ILB ASE. For more information on how to configure your app to have its own IP address, see Bind an existing custom SSL certificate to Azure web apps.
When an app has its own IP-based SSL address, the ASE reserves two ports to map to that IP address. One port is for HTTP traffic, and the other port is for HTTPS. Those ports are listed in the ASE UI in the IP addresses section. Traffic must be able to reach those ports from the VIP or the apps are inaccessible. This requirement is important to remember when you configure Network Security Groups (NSGs).
Network Security Groups
Network Security Groups provide the ability to control network access within a VNet. When you use the portal, there's an implicit deny rule at the lowest priority to deny everything. What you build are your allow rules.
In an ASE, you don't have access to the VMs used to host the ASE itself. They're in a Microsoft-managed subscription. If you want to restrict access to the apps on the ASE, set NSGs on the ASE subnet. In doing so, pay careful attention to the ASE dependencies. If you block any dependencies, the ASE stops working.
NSGs can be configured through the Azure portal or via PowerShell. The information here shows the Azure portal. You create and manage NSGs in the portal as a top-level resource under Networking.
When the inbound and outbound requirements are taken into account, the NSGs should look similar to the NSGs shown in this example. The VNet address range is 192.168.250.0/23, and the subnet that the ASE is in is 192.168.251.128/25.
The first two inbound requirements for the ASE to function are shown at the top of the list in this example. They enable ASE management and allow the ASE to communicate with itself. The other entries are all tenant configurable and can govern network access to the ASE-hosted applications.
A default rule enables the IPs in the VNet to talk to the ASE subnet. Another default rule enables the load balancer, also known as the public VIP, to communicate with the ASE. To see the default rules, select Default rules next to the Add icon. If you put a deny everything else rule after the NSG rules shown, you prevent traffic between the VIP and the ASE. To prevent traffic coming from inside the VNet, add your own rule to allow inbound. Use a source equal to AzureLoadBalancer with a destination of Any and a port range of \*. Because the NSG rule is applied to the ASE subnet, you don't need to be specific in the destination.
If you assigned an IP address to your app, make sure you keep the ports open. To see the ports, select App Service Environment > IP addresses.
All the items shown in the following outbound rules are needed, except for the last item. They enable network access to the ASE dependencies that were noted earlier in this article. If you block any of them, your ASE stops working. The last item in the list enables your ASE to communicate with other resources in your VNet.
After your NSGs are defined, assign them to the subnet that your ASE is on. If you don’t remember the ASE VNet or subnet, you can see it from the ASE portal page. To assign the NSG to your subnet, go to the subnet UI and select the NSG.
Forced tunneling is when you set routes in your VNet so the outbound traffic doesn't go directly to the internet but somewhere else like an ExpressRoute gateway or a virtual appliance. If you need to configure your ASE in such a manner then read the document on Configuring your App Service Environment with Forced Tunneling. This document will tell you the options available to work with ExpressRoute and forced tunneling.
When you create an ASE in the portal we also create a set of route tables on the subnet that is created with the ASE. Those routes simply say to send outbound traffic directly to the internet.
To create the same routes manually, follow these steps:
Go to the Azure portal. Select Networking > Route Tables.
Create a new route table in the same region as your VNet.
From within your route table UI, select Routes > Add.
Set the Next hop type to Internet and the Address prefix to 0.0.0.0/0. Select Save.
You then see something like the following:
After you create the new route table, go to the subnet that contains your ASE. Select your route table from the list in the portal. After you save the change, you should then see the NSGs and routes noted with your subnet.
Service Endpoints enable you to restrict access to multi-tenant services to a set of Azure virtual networks and subnets. You can read more about Service Endpoints in the Virtual Network Service Endpoints documentation.
When you enable Service Endpoints on a resource, there are routes created with higher priority than all other routes. If you use Service Endpoints with a forced tunneled ASE, the Azure SQL and Azure Storage management traffic isn't forced tunneled.
When Service Endpoints is enabled on a subnet with an Azure SQL instance, all Azure SQL instances connected to from that subnet must have Service Endpoints enabled. if you want to access multiple Azure SQL instances from the same subnet, you can't enable Service Endpoints on one Azure SQL instance and not on another. Azure Storage does not behave the same as Azure SQL. When you enable Service Endpoints with Azure Storage, you lock access to that resource from your subnet but can still access other Azure Storage accounts even if they do not have Service Endpoints enabled.