Deploy Azure Databricks in your Azure virtual network (VNet injection)
If you deployed an Azure Databricks workspace in your own Azure Virtual Network during the VNet injection preview and have not yet upgraded to GA, you will lose access to your workspace on June 1st. On or after June 1st, you must follow the upgrade steps and then open a support ticket to regain access to your workspace. See Upgrade your VNet Injection Preview Workspace to GA.
The default deployment of Azure Databricks is a fully managed service on Azure: all data plane resources, including a virtual network (VNet) that all clusters will be associated with, are deployed to a locked resource group. If you require network customization, however, you can deploy Azure Databricks data plane resources in your own virtual network (sometimes called VNet injection), enabling you to:
- Connect Azure Databricks to other Azure services (such as Azure Storage) in a more secure manner using service endpoints.
- Connect to on-premises data sources for use with Azure Databricks, taking advantage of user-defined routes.
- Connect Azure Databricks to a network virtual appliance to inspect all outbound traffic and take actions according to allow and deny rules.
- Configure Azure Databricks to use custom DNS.
- Configure network security group (NSG) rules to specify egress traffic restrictions.
- Deploy Azure Databricks clusters in your existing virtual network.
Deploying Azure Databricks data plane resources to your own virtual network also lets you take advantage of flexible CIDR ranges (anywhere between /16-/24 for the virtual network and up to /26 for the subnets).
You cannot replace the virtual network for an existing workspace. If your current workspace cannot accommodate the required number of active cluster nodes, we recommend that you create another workspace in a larger virtual network. Follow these detailed migration steps to copy resources (notebooks, cluster configurations, jobs) from the old to new workspace.
The virtual network that you deploy your Azure Databricks workspace to must meet the following requirements:
Location: The virtual network must reside in the same location as the Azure Databricks workspace.
Subscription: The virtual network must be in the same subscription as the Azure Databricks workspace.
Subnets: The virtual network must include two subnets dedicated to Azure Databricks: a private subnet and public subnet. The public subnet allows communication with the Azure Databricks control plane. The private subnet allows only cluster-internal communication. Do not deploy other Azure resources on the subnet used by your Azure Databricks workspace. Sharing the subnet with other resources, such as virtual machines, prevents managed updates to the intent policy for the subnet.
Azure Databricks will create these for you when you Create the Azure Databricks workspace in the Azure portal and will delegate your public and private subnet to the
Microsoft.Databricks/workspacesservice during workspace deployment, which allows Azure Databricks to create Network security group rules. Azure Databricks will always give advance notice if we need to add or update the scope of an Azure Databricks-managed NSG rule. Azure Databricks will also create subnets for you if you use the All-in-one Azure Resource Manager template or the VNet-only Azure Resource Manager template. However, if you use a custom Azure Resource Manager template or the Workspace Azure Resource Manager template, it is up to you to ensure that the subnets have network security groups attached and are properly delegated. For delegation instructions, see Upgrade your VNet Injection preview workspace to GA or Add or remove a subnet delegation.
There is a one-to-one relationship between these subnets and an Azure Databricks workspace. You cannot share multiple workspaces across a single subnet. You must have a new pair of public and private subnets for each workspace you deploy.
Address space: A CIDR block between /16 - /24 for the virtual network and a CIDR block up to /26 for the private and public subnets.
You can use the Azure Databricks workspace deployment interface in the Azure portal to automatically configure an existing virtual network with the required subnets, or you can use Azure-Databricks-supplied Azure Resource Manager templates to configure your virtual network and deploy your workspace.
This section describes how to create an Azure Databricks workspace in the Azure portal and deploy it in your own existing virtual network. Azure Databricks updates the virtual network with two new subnets and network security groups using CIDR ranges provided by you, whitelists inbound and outbound subnet traffic, and deploys the workspace to the updated virtual network.
If you want more control over the configuration of the virtual network—for example, you may want to use existing subnets, use existing network security groups, or create your own security rules—you can use Azure-Databricks-supplied Azure Resource Manager templates instead of the portal UI. See Configure the virtual network.
You must configure a virtual network to which you will deploy the Azure Databricks workspace:
You can use an existing virtual network or create a new one, but the virtual network must be in the same region and same subscription as the Azure Databricks workspace that you plan to create.
A CIDR range between /16 - /24 is required for the virtual network.
A workspace with a smaller virtual network can run out of IP addresses (network space) more quickly than a workspace with a larger virtual network. For example, a workspace with a /24 virtual network and /26 subnets can have a maximum of 64 nodes active at a time, whereas a workspace with a /20 virtual network and /22 subnets can house a maximum of 1024 nodes.
Your subnets will be created automatically when you configure your workspace, and you will have the opportunity to provide the CIDR range for the subnets during configuration.
In the Azure portal, select + Create a resource > Analytics > Azure Databricks or search for Azure Databricks and click Create or + Add to launch the Azure Databricks Service dialog.
Follow the configuration steps described in the Create an Azure Databricks workspace in your own Virtual Network quickstart.
Select the virtual network you want to use.
Name your public and private subnets and provide CIDR ranges in a block up to /26:
- A public subnet will be created with associated Network security group rules that allow communication with the Azure Databricks control plane.
- A private subnet will be created with associated network security group rules that allow cluster-internal communication.
- Azure Databricks will have delegated permissions to update both subnets via the
Microsoft.Databricks/workspacesservice. These permissions apply only to network security group rules that are required by Azure Databricks, not to other network security group rules that you add or to the default network security group rules included with all network security groups.
Click Create to deploy the Azure Databricks workspace to the virtual network.
When a workspace deployment fails, the workspace is still created in a failed state. Delete the failed workspace and create a new workspace that resolves the deployment errors. When you delete the failed workspace, the managed resource group and any successfully deployed resources are also deleted.
Advanced configuration using Azure Resource Manager templates
If you want more control over the configuration of the virtual network—for example, you want to use existing subnets, use an existing network security group, or add your own security rules—you can use the following Azure Resource Manager (ARM) templates instead of the portal-UI-based automatic virtual network configuration and workspace deployment.
If you deployed an Azure Databricks workspace in your own Azure Virtual Network during the VNet injection preview, you must upgrade your preview workspace to the GA version before March 31, 2020. The primary upgrade task is to delegate your public and private subnets to the
Microsoft.Databricks/workspaces service, which allows Azure Databricks to create Network security group rules. Azure Databricks always gives advance notice if we need to add or update the scope of an Azure Databricks-managed NSG rule. See Upgrade your VNet Injection preview workspace to GA.
If you used Azure Resource Manager templates to deploy an Azure Databricks workspace to your own virtual network during the preview, and you want to continue to use Azure Resource Manager templates to create virtual networks and deploy workspaces, you should use the upgraded Azure Resource Manager templates listed below. Follow the template link to get the latest version.
If you are using a custom Azure Resource Manager template or the Workspace Template for Databricks VNet Injection to deploy a workspace to an existing virtual network, you must create public and private subnets, attach a network security group to each subnet, and delegate the subnets to the
Microsoft.Databricks/workspaces service before deploying the workspace. You must have a separate pair of public/private subnets for each workspace that you deploy.
To create a virtual network and Azure Databricks workspace all in one, use the All-in-one Template for Databricks VNet Injected Workspaces.
To create a virtual network with the proper public and private subnets, use the Virtual Network Template for Databricks VNet Injection.
To deploy an Azure Databricks workspace to an existing virtual network, use the Workspace Template for Databricks VNet Injection.
Your virtual network’s public and private subnets must have network security groups attached and must be delegated to the
Microsoft.Databricks/workspaces service before you use this Azure Resource Manager template to deploy a workspace. You can use the Virtual Network Template for Databricks VNet Injection to create a virtual network with properly delegated subnets. If you are using an existing virtual network and you have not delegated the public and private subnets, see Add or remove a subnet delegation or Upgrade your VNet Injection preview workspace to GA.
You must have a separate pair of public/private subnets for each workspace that you deploy.
The following tables display the current network security group rules used by Azure Databricks. If Azure Databricks needs to add a rule or change the scope of an existing rule on this list, you will receive advance notice. This article and the tables will be updated whenever such a modification occurs.
Network security group rules for workspaces created after January 13, 2020
The following table only applies to Azure Databricks workspaces created after January 13, 2020. If your workspace was created prior to January 13, 2020, see the next table.
|Direction||Protocol||Source||Source Port||Destination||Dest Port||Used|
|Inbound||TCP||AzureDatabricks (service tag)||Any||VirtualNetwork||22||Public IP|
|Inbound||TCP||AzureDatabricks (service tag)||Any||VirtualNetwork||5557||Public IP|
|Outbound||TCP||VirtualNetwork||Any||AzureDatabricks (service tag)||443||Default|
Network security group rules for workspaces created before January 13, 2020
The following table only applies to Azure Databricks workspaces created before January 13, 2020. If your workspace was created on or after January 13, 2020, see the previous table.
|Direction||Protocol||Source||Source Port||Destination||Dest Port||Used|
|Inbound||TCP||ControlPlane IP||Any||VirtualNetwork||22||Public IP|
|Inbound||TCP||ControlPlane IP||Any||VirtualNetwork||5557||Public IP|
Azure Databricks is a Microsoft Azure first-party service that is deployed on the Global Azure Public Cloud infrastructure. All communications between components of the service, including between the public IPs in the control plane and the customer data plane, remain within the Microsoft Azure network backbone. See also Microsoft global network.
Workspace creation errors
Possible cause: you are creating a workspace in a VNet whose private and public subnets have not been delegated to the
Microsoft.Databricks/workspaces service. Each subnet must have a network security group attached and must be properly delegated. See Virtual network requirements for more information.
Possible cause: you are creating a workspace in a VNet with private and public subnets that are already being used by an existing Azure Databricks workspace. You cannot share multiple workspaces across a single subnet. You must have a new pair of public and private subnets for each workspace you deploy.
Cluster creation errors
Instances Unreachable: Resources were not reachable via SSH.
Possible cause: traffic from control plane to workers is blocked. If you are deploying to an existing virtual network connected to your on-premises network, review your setup using the information supplied in Connect your Azure Databricks Workspace to your on-premises network.
Unexpected Launch Failure: An unexpected error was encountered while setting up the cluster. Please retry and contact Azure Databricks if the problem persists. Internal error message:
Timeout while placing node.
Possible cause: traffic from workers to Azure Storage endpoints is blocked. If you are using custom DNS servers, also check the status of the DNS servers in your virtual network.
Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster. See the Azure Databricks guide for more information. Azure error code:
Possible cause: the virtual network or subnets do not exist any more. Make sure the virtual network and subnets exist.
Cluster terminated. Reason: Spark Startup Failure: Spark was not able to start in time. This issue can be caused by a malfunctioning Hive metastore, invalid Spark configurations, or malfunctioning init scripts. Please refer to the Spark driver logs to troubleshoot this issue, and contact Databricks if the problem persists. Internal error message:
Spark failed to start: Driver failed to start in time.
Possible cause: Container cannot talk to hosting instance or DBFS storage account. Fix by adding a custom route to the subnets for the DBFS storage account with the next hop being Internet.