My Personal FAQ on Azure Networking and ARM
Last updated: July 24, 2016.
IMPORTANT : The information I reported in this post is time sensitive since may change in the future. I will try to maintain the content updated, but you should double--check with official Azure documentation sources. This is a new article specifically focused on ARM, you can find a previous old one related to ASM at the link below:
My personal Azure FAQ on Azure Networking SLAs, bandwidth, latency, performance, SLB, DNS, DMZ, VNET, IPv6 and much more
In the last four years I worked extensively on Azure IaaS, during engagements with my partners I have been asked many questions on Azure networking in general, and more specifically on Virtual Network (VNET) and VMs in new Azure Resource Manager (ARM) world. Providing adequate answers is not always easy, sometime the documentation is not clear or missing in details, while other times you can deduct yourselves but requires pretty good knowledge. Since I detected some recurring patterns in these questions, I decided to write my personal FAQ list in this blog post. Just to be clear: even if I’m a MSFT and have inner knowledge of Azure, I’m not going to unveil you any reserved or secret information. Everything you will read here can be found playing directly with Azure or retrieved by using official and public documentation. If you have additional “nice” questions requiring a non-trivial or non-easy-to-retrieve answer, fill free to insert a comment below this post. Additionally, if you have proposals or feedbacks regarding new feature that Azure networking should have, please use the link below to submit your ideas or vote existing ones:
FEEDBACK FORUM: Networking (DNS, Traffic Manager, VPN, VNET)
As usual, you can also follow me on Twitter at @igorpag. Regards.
Can I create a Storage Account or Azure SQLDB server without a public VIP?
While is possible to create a Virtual Machine not exposed to the Internet, for Azure Storage Accounts and Azure SQLDB logical servers there is a mandatory DNS name created that need to be exposed. During creation of these resources, a FQDN records is created and exposure cannot be avoided since this resource locator is required to access the resource. Both these resources are protected by Microsoft DDoS and Azure SQLDB can be also configured with its own firewall mechanism.
Configure an Azure SQL Database server-level firewall rule using the Azure Portal
Do Azure VPN or ExpressRoute provide traffic encryption?
Azure VPN tunnels, for Site-to-Site connectivity (S2S), are by design/default encrypted using IPSEC. Encryption is also provided for Point-to-Site (P2S) using SSTP (Secure Socket Tunneling Protocol). ExpressRoute does not provide network traffic encryption for its circuits.
VPN Gateway FAQ
Do Azure VPN or ExpressRoute provide QoS?
ExpressRoute provides network traffic QoS only for Skype for Business, while Azure VPN does not provide this kind of service. You can read more details on ExpressRoute at the link below:
ExpressRoute QoS requirements
Do Azure VPN and ExpressRoute support BGP?
YES, now also Azure VPN supports Border Gateway Protocol (BGP) for automatic exchange of routing information when different VNETs and/or on-premises sites are connected together. ExpressRoute fully support BGP since the beginning. Transit routing is supported, but redundant tunnels between a pair of virtual networks are not supported yet. Be aware that is not possible today to filter BGP routes at the VPN level, if you need to build more advanced network topology, you should give a look to the second URL listed below.
Overview of BGP with Azure VPN Gateways
Hub&Spoke, Daisy-Chain and Full-Mesh VNET topologies in Azure ARM (V2)
Can I connect two Azure Virtual Networks without any Gateway?
This feature is coming, stay tuned! It will permit to connect different Azure VNETs without VPN or ExpressRoute, then without bandwidth caps imposed by the Gateway.
Can I assign multiple IPs to my single Network Interface Card (NIC)?
This feature is coming, stay tuned!
Is it possible to change subnet or virtual network for my existing Virtual Machine?
With some limitations, it is possible to change subnet for an existing Virtual Machine, but changing Virtual Network (VNET) is not directly possible, you will need to export VM definition/disks and recreate in the new VNET. You can read more details in the article below:
How to change Subnet and Virtual Network for Azure Virtual Machines (ASM & ARM)
Can I mix ASM ARM Virtual Machines and Virtual Networks?
NO, only Virtual Machines created with Azure Resource Manager (ARM) API can join ARM Virtual Networks, similarly only Azure Service Manager (ASM) Virtual Machines can join ASM Virtual Network. Additionally, migration is possible from ASM VMs/VNETs and Azure Resource Manager resource model, but this will happen at the VNET level, then including all the contained VMs at the same time. You can read more about migration tools and procedures at the link below:
Technical deep dive on platform-supported migration from classic to Azure Resource Manager
Does Azure Load Balancer support multiple IPs for the same set of Virtual Machines?
YES, now also using Azure Resource Manager (ARM), it is possible to have multiple IPs assigned to the same load balancer and then traffic forwarded to the same set of Virtual Machines included in one single Availability Set. This feature was previously available using the Azure Service Manager (ASM) model. For a code sample on how to add multiple VIP to Azure SLB, you can read the article below:
Adding additional VIP to Azure Load Balancer
Does Azure Load Balancer provide Layer-7 load-balancing?
Azure built-in load balancer is Layer-4 (TCP/UDP), but if you need Layer-7 load-balancing, you can use Azure Application Gateway or one of the Network Virtual Appliance (NVA) available in the Marketplace by 3rd-party vendors.
Azure Load Balancer Overview
Can I customize the Internet facing FQDN used by Azure for my public IP?
Without using vanilla domains with CNAME records, you can only partially change the FQDN associated to your public IP, doesn’t matter if reserved or not, doesn’t matter if VIP or Instance-Level IP (ILPIP, PIP). The suffix will be always in the form of “location.cloudapp.azure.com” , where “location” is the Azure region where you deployed, you can only add a “domainnamelabel” prefix and obtain “domainnamelabel .location.cloudapp.azure.com” DNS domain.
IP addresses in Azure
Can I use Receive Side Scaling (RSS) optimization in Azure VM?
YES. Azure HyperV is able to support RSS on the host side, but to leverage this functionality in Guest OS you need to manually enable it following the instructions at the link below:
Virtual Receive-side Scaling in Windows Server 2012 R2
For Windows Server, Guest OS must be Windows Server 2012 R2 version or later, while for support on Linux OS, see the article below:
Linux and FreeBSD Virtual Machines on Hyper-V
Does ICMP work in Azure?
It depends from where you want to use ICMP tools and which targets you want to reach. ICMP works inside an Azure Virtual Network, between different Azure Virtual Networks connected via VPN or ExpressRoute or hybrid connections between on-premise and Azure VNET. It will *not* work if you have to cross the Azure Load Balancer (SLB), for example if you want to ping a Cloud Service from another Cloud Service. If you need ICMP for diagnostic and troubleshooting, there are valid alternatives to PING and TRACERT not based on ICMP but TCP. My favorite ones, among many others available in Internet, are PSPING from Mark Russinovich SysInternals tools and NMAP:
Does Azure provide precise numbers on VM network bandwidth?
I tested VM bandwidth limits on almost all VM sizes and found the results pretty consistent over time, then in theory there is nothing preventing you from generating your measure. What Azure documentation provide is a relative measure of network capacity for each VM SKU/SIZE as you can see in the table below:
Sizes for virtual machines in Azure
If you look at column “Max Network Bandwidth”, you can find tags as LOW, MODERATE, HIGH, VERY HIGH, EXTREMELY HIGH. If you found in Internet, some old tables and information related to maximum network bandwidth for various VM SKUs, you should not trust them since outdated and Azure infrastructure evolved in the meanwhile. The only exceptions are Azure HPC instances where it is exactly specified the type of network adapter provided, that is 1 NIC 32Gbit/s and 1 NIC 10Gbit/s for A8/A9, and 1 NIC 10Gbit/s for A10/A11. See at the link below for more information:
About the A8, A9, A10, and A11 compute-intensive instances
Does Azure provide SLA on VM-to-VM network latency?
During my tests and projects, Azure network Always provided excellent performances in terms of latency. At least at my knowledge, there is no Cloud vendor able to guarantee a maximum network latency, backed up by a formal SLA, between two different VMs. It’s very easy to test yourself and have a good understanding of average latencies, but you need to be very careful on your test criteria. First of all, you need to test over various hours during the day including peak and non-peak hours, various days over the week including weekends and working days, then finally you should use calculate the average and consider a good percentile sampling near the 95% to eliminate peaks and occasional strange values. Additionally, don’t assume latencies will be the same in all Azure datacenters. If you want to do your own tests, I would recommend you PSPING tool from SYSINTERNAL suite that you can find here: http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx. What is really nice regarding this tool, is that it will not use ICMP then no problem to traverse firewalls and load-balancers, you can decide which TCP port to use and, very important, you can test for bandwidth, not only for latency. You can read full details on using this tool to test Azure latency in one of my previous blog post:
Azure Network Latency & SQL Server Optimization
In the past, “Affinity Group” concept has been used to provide co-location of VMs and minimize latency: now this concept is officially deprecated with the new Azure Resource Manager (ARM) API and broad adoption of “Regional” VNET. Azure infrastructure continuous evolutions made network more efficient and performant, then you should not use it anymore.
Azure Datacenter-to-Datacenter traffic will go through public Internet?
Traffic between VNETs in different datacenters, connected via VPN connection will remain inside Microsoft owned network backbones, see slide  in the presentation below:
Introduction to Microsoft Azure Networking Technologies and What's New
Also in the case of VNETs connected using Express Route, traffic will not go through public Internet, private leased line will be used.
Is it possible to know Azure public IP address pools?
YES, all Azure public IP ranges are published at the link below. If you want to white-list, for security reasons, on your on-premise network you can do that selectively by region but not by service.
Microsoft Azure Datacenter IP Ranges
If you rely on specific Azure resources, for example an Azure SQLDB instance or blob storage account, be aware that the assigned IP you will see, resolving their specific DNS names, may change without notice. IMPORTANT: don’t rely on any geo-location service for Internet IP used by Azure since may report incorrect information, for more details see the blog post below:
Microsoft Azure’s use of non-US IPv4 address space in US regions
If you created your own Cloud Service, and want to be possible to validate it using reverse-DNS, you can do that enabling PTR record registration (by Azure) for your Cloud Service VIP as described at the link below:
Announcing: Reverse DNS for Azure Cloud Services
Be aware that the Virtual IP (VIP) associated to your Cloud Service may change under some conditions, if you need to be absolutely sure it will never happen, you need to enable "Reserved IP" feature as described at the blog post below:
Reserved IP addresses for Cloud Services & Virtual Machines
Is there any API to retrieve Azure public IP addresses?
NO, there is no built-in API provided by the platform at the moment, instead you can download the XML file from the link below. Or you can build a simple logic to download the file and parse content as shown in the second URL listed below:
Microsoft Azure Datacenter IP Ranges
Step-by-Step: Automate Building Outbound Network Security Groups Rules via Azure Resource Manager (ARM) and PowerShell
Azure “Affinity Group” is officially deprecated?
YES, with the introduction of Azure Resource Manager (ARM) new API model, “Affinity Group” (AG) is officially deprecated, thus not recommended to use it.
Azure Subscription and Service Limits, Quotas, and Constraints
Even if AG can be still used in Service Management (SM) API, but not in the legacy and new Azure portals, usage is not recommended since many Azure networking features rely on “Regional Virtual Networks”. Using an AG to pin a Virtual Network in fact, will originate a “Local Network” tied to a single Azure Cluster and will not have benefits of the new Azure scale units for compute resources.
Regional Virtual Networks
What is “*.internal.cloudapp.net” DNS suffix in Azure VM?
If you build an Azure VM and join to an Azure Virtual Network and try to ping it using the VM host name (legacy Netbios name to be clear), this is what you will see:
As you can see in the picture, there are two strange parts in the FQDN: nothing strange in “cloudapp.net”, but what are “internal” and “a3”? You can easily guess that “internal” is the default DNS sub-zone that Azure internal DNS (iDNS) will use to host records for VMs and resolve to internal VM IP (DIP). “a3” instead is a little bit complex: in short, it’s describing the network zone, inside the specific Azure datacenter, where the VM is allocated. There is no official documentation or list of these zones, then don’t ask more, but please note that it will change not only between Azure datacenters, but also inside a single datacenter. You can programmatically retrieve internal DNS setting using the simple PowerShell command below:
(Get-AzureRmNetworkInterface -Name "mynicname" -ResourceGroupName $rgname).DnsSettings.InternalFqdn
Output will be something like the string below:
InternalDomainNameSuffix : 0fcdtenvgl4urm0sv5aj1xkcrc.fx.internal.cloudapp.net
PING and TRACERT tools will work inside Azure Virtual Network?
YES, they will work perfectly inside a VNET, and even from on-premise to VNET via VPN connection. They will not work if you try to cross the Azure Load Balancer (SLB/ILB).
Azure Load Balancer (SLB) uses Round-Robin policy to distribute incoming connections?
This is a very common mis-understanding on how SLB works. Actually, Azure SLB is a Layer-4 software Load Balancer and uses a 5-tuple (source IP, source port, destination IP, destination port, protocol type) to calculate the hash that is used to map traffic to the available servers behind a VIP. The hash function is chosen such that the distribution of connections to servers is fairly random then not perfect round-robin. Additionally, at least today, session affinity is not supported.
Microsoft Azure Load Balancing Services
More recently, Azure introduced a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). Azure Load Balancer can be configured to use a 2 tuple (Source IP, Destination IP) or 3-tuple (Source IP, Destination IP, Protocol) to map traffic to the available servers. By using Source IP affinity, connections initiated from the same client computer goes to the same DIP endpoint.
Azure Load Balancer new distribution mode
Azure Load Balancer (SLB) supports SSL termination?
Azure SLB (or ILB) does not support SSL termination at the edge, but you can use Azure Application Gateway to achieve this goal:
Application Gateway Overview
Alternatively, you can handle the termination process for HTTPS encryption and decryption within each VM or Web/Worker role instance, use some 3rd-party appliances or OSS technologies as NGINX or HAPROXY.
Can I use my own Load Balancer and avoid Azure Load Balancer?
YES, usage of Azure SLB is not mandatory, there is a specific offer from Azure called “Basic” that will give you up to 27% lower prices, compared to Standard SKUs:
Basic Tier Virtual Machines
Autoscaling will not be available with “Basic”, but you will still have 99,95% through “Availability Set”. Please also notes that not all the sizes, in terms of CPUs and RAM, will be available:
Once selected the “Basic” tier for your VM, you can then install your own Load Balancer software. Be aware that sizes available in the “Basic” tiers are very limited and all listed in the picture and article above.
Can I expose full port range of my VM?
YES, there is an Azure specific feature called “Instance-Level Public IP Addresses” that will provide your VM a “Public Instance IP” (PIP) with full port range opened by default, without any need to create any endpoint. Using normal VIPs, you can only create/open up to 150 endpoints/ports, then if you need a large number of ports to be opened this is the feature that will satisfy your requirements.
Instance-Level Public IP Addresses
How many connections a VM can support?
For Windows Server, a VM can support about 500k TCP connections but you need to be careful about other potential limits that may come into the game before this threshold: if you expose your VM to Internet traffic through an Azure Load Balancer endpoint, you may be limited by SLB capacity or DDOS security mechanisms. Actually, there is no public documentation on SLB or Endpoint limits, neither on the number of connection nor network bandwidth, then you should conduct your own performance tests to ensure everything will work correctly.
Can I have multiple VNET Gateways?
Currently you can only have one virtual network Gateway per single VNET per single type, that is one for VPN and one for ExpressRoute. With VPN, if you configure multiple tunnels, they will share the same maximum network bandwidth.
ExpressRoute and VPN can coexist in the same VNET?
YES, it is possible to have a VNET with gateways supporting ExpressRoute and VPN at the same time. The only requirement is to enable ExpressRoute before VPN. One Gateway per VNET restriction persists.
New Networking Capabilities for a Consistent, Connected and Hybrid Cloud
Can I connect VNETs across regions, geographies and subscriptions?
YES, Azure provides two ways to connect VNETs: ExpressRoute and VPN. With VPN, there is no boundary on the region, geography or subscription, essentially the tunnel only need Gateway VIPs to see each other. In case you need more bandwidth and lower latency, you can eventually use ExpressRoute technology to connect up to 100 VNETs (depending on bandwidth) and share circuits between them, even with different subscriptions. Before Premium add-on announced in May 2015, ExpressRoute posed limitations on geographical boundaries: now it is possible not only to connect VNETs in the same geography (US, Europe, etc.) but also across different geographies. For example, you can connect a VNET in West US to a VNET in North Europe.
New Networking Capabilities for a Consistent, Connected and Hybrid Cloud
Does Azure provide DDoS network protection?
YES, Azure infrastructure is designed to protect the network from DDOS originating from the Internet and also internally from other tenants VMs, you can read the details in the white-paper below:
Microsoft Azure Network Security Whitepaper version 3 is now available
Please be aware of the following important points:
- Windows Azure’s DDoS defense system is designed not only to withstand attacks from the outside, but also from within.
- Windows Azure monitors and detects internally initiated DDoS attacks and removes offending VMs from the network.
- Windows Azure’s DDoS protection also benefits applications. However, it is still possible for applications to be targeted individually. As a result, customers should actively monitor their Windows Azure applications.
To augment your tenant security, you can deploy 3rd-party Network Virtual Appliances (NVA) in your Virtual Network, but be aware that custom DDoS to protect Azure public IPs (VIPs) is not supported.
Can I use IPv6 inside an Azure Virtual Network?
IPv6 support has been confirmed to be under development as you can read at the link below:
Support IPv6 throughout the Azure Platform
Additional comments from Azure FAQ on http://azure.microsoft.com/en-us/pricing/faq:
Microsoft has played a leading role in helping customers to smoothly transition from IPv4 to IPv6 for the past several years. To date, Microsoft has built IPv6 support into many of its products and solutions like Windows 8 and Windows Server 2012 R2. Microsoft is committed to expanding the worldwide capabilities of the Internet through IPv6 and enabling a variety of valuable and exciting scenarios, including peer-to-peer and mobile applications. The foundational work to enable IPv6 in the Azure environment is well underway. However, we are unable to share a date when IPv6 support will be generally available at this time.
UDP broadcast and multicast are supported inside Azure Virtual Network?
This type of communication is not allowed inside a VNET or even across Azure SLB.
Support Multicast within Virtual Networks
Is it possible to have multiple NICs on VM?
YES, this feature has been announced at TechEd Europe back in October 2014. Be aware that adding more NICs will not give you more bandwidth, all will share the same limit granted to the VM level. Now it's possible to have up to 4 additional NICs without any additional cost, the exact number varies depending on the VM size:
- Large (A3) and A6: 2
- ExtraLarge (A4) and A7: 4
- A9: 2
- D3: 2
- D4: 4
- D13: 4
Be aware that some limitations exist, be sure to read the “Multiple NIC FAQs & Constraints” section in the blog post below:
Multiple VM NICs and Network Virtual Appliances in Azure
Can I persist my VM network card MAC address?
Work is underway to provide this feature, that is preserving MAC addresses across Azure Host failures, VM resizes and re-deployments.
Can I avoid DHCP for my VM inside Virtual Network?
YES, when you join a VM to an Azure Virtual Network, by default it will take an internal IP (DIP) automatically assigned by Azure internal DHCP, valid in the address scope you defined and eventually in the subnet you indicated. This IP will have an infinite lease time, but under certain conditions may change, for example if you stop your VM with de-provisioning (default behavior). If you are going to mix DHCP dynamically assigned IPs and statically assigned IPs, it’s recommended to use separate subnets to avoid potential address clashes. You can read more details in the blog post below:
Static Internal IP Address for Virtual Machines
Is it possible to run network penetration tests against my Azure VMs?
YES, you can do that but it’s highly recommended to follow the specific procedure mentioned below, before running the test, otherwise Azure monitoring and defense systems will trigger in and blacklist your connections, IPs and/or VMs. Be sure your request will be approved before starting the test, you can find the form to fill-in at the bottom of the page URL reported below:
Penetration Test Notification Process https://security-forms.azure.com/penetration-testing/terms
Microsoft conducts regular penetration testing to improve Azure security controls and processes. We understand that security assessment is also an important part of our customers' application development and deployment. Therefore, we have established a policy for customers to carry out authorized penetration testing on their applications hosted in Azure. Because such testing can be indistinguishable from a real attack, it is critical that customers conduct penetration testing only after notifying Microsoft. Penetration testing must be conducted in accordance with our terms and conditions.
Which is the guaranteed bandwidth for Azure VPN?
There is no minimum guaranteed bandwidth for Azure VPN, the only SLA provided is about high-availability (99,90%). Azure provides an indication for maximum theoretical limit that actually is 200Mbit/sec with “High-Performance” SKU:
About VPN Gateway
As you can see in the table above, there is a limit on the maximum number of tunnels you can create, based on the type of Gateway SKU and connectivity type (S2S VPN or Express Route).
VHD I/O will count toward VM network bandwidth cap?
I/O related to IaaS VM VHDs access will count against storage IOPS limits, not network limits. Seems a tricky question, but since persistent VM storage is networked, some customers and partners frequently ask it. Official numbers on scalability targets for Azure Storage are reported at the link below:
Azure Storage Scalability and Performance Targets
If you use some special features like SQL Server 2014 Azure Blob Storage integration or Azure Files (over SMB), it will count toward your VM network bandwidth limits, in addition to the target service scalability and performance limits.
UPDATED: New White-Paper on SQL Server 2014 and Azure Blob storage integration
Introducing Microsoft Azure File Service
Can I use A8/A9 VM Infiniband NIC for my application traffic?
It depends. I have been asked several times if, for example, Infiniband NIC can be used for SQL Server AlwaysOn Availability Group replication traffic, since it provides very high-bandwidth, low-latency connection and RDMA support. In this specific case, the answer is NO, this NIC does not provide general TCP/IP connectivity, as for all the applications that cannot talk over “Network Direct” interface and MPI protocol.
New High Performance Capabilities for Windows Azure
Can I create a DMZ in Azure?
YES. With the introduction of “Network Security Groups” (NSG), Network Virtual Appliances (NVA), “Multiple NICs”, “User Defined Routes” and “IP Forwarding” it is now possible to implement a full-featured DMZ with strong segregation of traffic and full control between subnets inside a VNET. Creating a DMZ requires careful planning and architectural work, you can find a guided step-by-step tour, along with examples, at the link below:
Microsoft cloud services and network security
There is already a certain number of 3rd-party Network Virtual Appliances (NVAs) you can use to provide security features like firewalls/DMZ, but also IDS/IPS, you can find them in the Azure Market Place through Azure Certified Program to ensure quality and simplify deployment.
Azure ILB can be used to augment networking security?
YES, but it’s necessary to clarify what is the security benefit since there is some confusion on how this feature works. Essentially, it can help security since will let you define one (or more) load-balanced endpoints, as when using Azure SLB, not exposed to the Internet, then more secure. Conversely, it’s not an isolation mechanism since will not segregate resources inside a VNET, all the VMs will have open and full connectivity to all other VMs if not using ILB, even if ACLs will be used. For the same reason, ILB alone is not enough to build a DMZ.
If you want to build a real DMZ configuration, it is recommended to look at the previous “Can I create a DMZ in Azure?” section in this blog post.