Upgrade your VNet Injection Preview Workspace to GA

With the ability to deploy an Azure Databricks workspace in your own Azure Virtual Network (sometimes called VNet injection) now transitioned from preview to general availability, you should upgrade your preview workspace to the GA version. To maintain support for your VNet injection workspace deployment, you must complete this upgrade by January 31, 2020, unless you are told otherwise by your Azure Databricks representative.

In the GA version of VNet injection, unlike the preview version, Azure Databricks manages all network security group (NSG) rules that are required by the Azure Databricks deployment. For this reason, the upgrade process involves delegating your public and private subnets to the Microsoft.Databricks/workspaces service, which allows Azure Databricks to maintain those network security group rules. This delegation does not give Azure Databricks any rights to update network security group rules you may add to the subnets yourself.

This process will not interfere with your existing Azure Databricks clusters or running jobs, and will make no visible changes to your Azure Databricks workspace.

Prerequisites

You must have the following permission: Microsoft.Network/virtualNetworks/subnets/write. Users with the Owner or Contributor role have this permission by default. To learn how to assign this permission, see Permissions.

Upgrade using Azure CLI

  1. Log in to the Azure CLI.

    az login
    
  2. Set environment variables.

    subscriptionId=<Your Subscription ID>
    vnetName=<Your Virtual Network’s Name>
    rgName=<Your Virtual Network’s Resource Group>
    publicSubnetName=<Name of Your Virtual Network’s Public Subnet>
    privateSubnetName=<Name of Your Virtual Network’s Private Subnet>
    delegation='Microsoft.Databricks/workspaces'
    
  3. Delegate the public subnet to Azure Databricks.

    az network vnet subnet update --subscription $subscriptionId --resource-group $rgName --vnet-name $vnetName --name $publicSubnetName --delegation $delegation
    
  4. Delegate the private subnet to Azure Databricks.

    az network vnet subnet update --subscription $subscriptionId --resource-group $rgName --vnet-name $vnetName --name $privateSubnetName --delegation $delegation
    

Upgrade using PowerShell

  1. Install the networking module.

    Install-Module -Name Az.Network -AllowClobber -Force
    
  2. Set environment variables.

    $subscriptionId = <Your Subscription ID>
    $vnetName = <Your Virtual Network Name>
    $rgname = <Your Virtual Network's Resource Group>
    $delegation = 'Microsoft.Databricks/workspaces'
    $publicSubnetName = <Name of Your Virtual Network’s Public Subnet>
    $privateSubnetName = <Name of Your Virtual Network’s Private Subnet>
    
  3. Set the subscription in your shell.

    Select-AzSubscription -SubscriptionId $subscriptionId
    
  4. Retrieve your virtual network and corresponding subnets.

    $vNet = Get-AzVirtualNetwork -Name $vnetName -ResourceGroupName $rgname
    $publicSubnet = Get-AzVirtualNetworkSubnetConfig -name $publicSubnetName -VirtualNetwork $vNet
    $privateSubnet = Get-AzVirtualNetworkSubnetConfig -name $privateSubnetName -VirtualNetwork $vNet
    
  5. Create a new delegation to Azure Databricks.

    $delegation = New-AzDelegation -Name adbDelegation -ServiceName "Microsoft.Databricks/workspaces"
    
  6. Set your public and private subnets to the new delegation and update the virtual network.

    Set-AzVirtualNetworkSubnetConfig -Name $publicSubnet.Name -VirtualNetwork $vNet -Delegation $delegation -AddressPrefix $publicSubnet.AddressPrefix
    
    Set-AzVirtualNetworkSubnetConfig -Name $privateSubnet.Name -VirtualNetwork $vNet -Delegation $delegation -AddressPrefix $privateSubnet.AddressPrefix
    
    Set-AzVirtualNetwork -VirtualNetwork $vNet
    

Upgrade using the Azure portal

  1. In the Azure portal, navigate to the virtual network where your Azure Databricks workspace is deployed. See View virtual networks and settings.

    no-alternative-text

  2. In the left menu, click Subnets. You’ll see your private and public subnet information displayed.

    no-alternative-text

  3. Click the public subnet row, go to the Subnet delegation dropdown, and select the Microsoft.Databricks/workspaces service.

    no-alternative-text

    For more information about subnet delegation, see Add or remove a subnet delegation.

  4. Repeat the subnet delegation for the private subnet.

  5. Save your changes.

Upgrade using Azure Resource Manager templates

Important

If you used Azure Resource Manager (ARM) templates to deploy a Azure Databricks workspace to your own virtual network during the preview, and you want to continue to use Azure Resource Manager templates to create virtual networks and deploy workspaces, you should use the upgraded Azure Resource Manager templates. See Configure the virtual network.

Post-upgrade steps

Once you have completed the subnet delegation, Azure Databricks will complete your workspace upgrade within one week. You will receive a notification of completion via Azure Communications. If your upgrade fails, you will also receive a notification. When your upgrade is complete, you should see a new set of network security rules in the network security group attached to your public and private subnets. Each of these rule names starts with the prefix Microsoft.Databricks-workspaces. Any rules that begin with the prefix databricks are no longer necessary, and you should delete them using the following procedure:

  1. In the Azure portal, navigate to the virtual network where your Azure Databricks workspace is deployed. See View virtual networks and settings.

    no-alternative-text

  2. In the left menu, click Subnets, and copy the name of the network security group for both your private and public subnets.

    no-alternative-text

  3. Paste the public subnet’s network security group name into the Search bar to open the Network security group Overview page.

  4. On the Overview page, find all inbound and outbound rules that start with “databricks” and delete them.

    no-alternative-text

  5. Repeat the previous two steps for the private subnet.