Enable Azure Private Link

Important

This feature is in Public Preview.

This article explains how to use Azure Private Link to enable private connectivity between users and their Databricks workspaces, and also between clusters on the data plane and the core services on the control plane within the Databricks workspace infrastructure.

Important

This article mentions the term data plane, which is the compute layer of the Azure Databricks platform. In the context of this article, data plane refers to the Classic data plane in your Azure subscription. By contrast, the Serverless data plane that supports Serverless SQL warehouses (Public Preview) runs in the Azure subscription of Azure Databricks. To learn more, see Serverless compute.

Overview

Private Link provides private connectivity from Azure VNets and on-premise networks to Azure services without exposing the traffic to the public network. Azure Databricks supports the following Private Link connection types:

  • Front-end Private Link, also known as user to workspace: A front-end Private Link connection allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint. The front-end connection is also used by JDBC/ODBC and PowerBI integrations. The network traffic for a front-end Private Link connection between a transit VNet and the workspace control plane traverses over the Microsoft backbone network.

  • Back-end Private Link, also known as data plane to control plane: Databricks Runtime clusters in a customer-managed VNet (the data plane) connect to a Azure Databricks workspace’s core services (the control plane) in the Azure Databricks cloud account. This enables private connectivity from the clusters to the secure cluster connectivity relay endpoint and REST API endpoint.

  • Web auth private connections: To support private front-end connections, special configuration is required to support the single sign-on (SSO) login callbacks to the Azure Databricks web application. A special type of private connection with sub-resource type browser_authentication hosts a private connection from the transit VNet that allows Azure Active Directory to redirect users after login to the correct control plane instance. One of these connections is shared for all workspaces in the region.

    Important

    Databricks strongly recommends creating a workspace called a private web auth workspace for each region to host the web auth private network settings. This workspace must not be deleted. All other workspaces in that region must refer to the Web auth private connection configuration of this workspace. This solves the problem of deleting a workspace potentially affecting other workspaces in that region. For details, see Step 4: Configure DNS to support SSO authentication flow (required for UI access).

You can implement both front-end and back-end Private Link connections or just the back-end connection. This article discusses how to configure both Private Link connection types.

If you implement Private Link for both front-end and back-end connections, you can optionally mandate private connectivity for the workspace, which means Azure Databricks rejects any connections over the public network. If you decline to implement both front-end or back-end connection types, you cannot enforce this requirement.

Important

Be sure that you understand the Private Link configuration options before deploying your workspace and other cloud resources. You can set up Private Link connectivity with a new workspace, but you cannot update the workspace fields associated with Private Link front-end and back-end connections on a running workspace.

The following table describes important terminology.

Terminology Description
Azure Private Link An Azure technology that provides private connectivity from Azure VNets and on-premise networks to Azure services without exposing the traffic to the public network.
Azure Private Link service A service that can be the destination for a Private Link connection. Each Azure Databricks control plane instance publishes an Azure Private Link service.
Azure private endpoint An Azure private endpoint enables a private connection between a VNet and a Private Link service. For front-end and back-end connectivity, the target of a Azure private endpoint is the Azure Databricks control plane.

For general information about private endpoints, see the Microsoft article What is a private endpoint?.

The following diagram shows the network flow in a typical implementation.

PL network architecture

Requirements

Azure subscription

  • Your Azure workspace must be on the Premium tier.

Azure Databricks workspace

Important

You cannot update an existing workspace to change certain workspace attributes that relate to Private Link:

  • You cannot update a workspace with the default (Databricks-managed) VNet and change it to use VNet injection.
  • You cannot update a workspace that does not use secure cluster connectivity to enable secure cluster connectivity.
  • You cannot update a workspace to modify its Private Link fields Required Network Access or Required NSG Rules.

Network architecture

  • You need a VNet that satisfies the requirements of VNet injection.

    • As discussed in that article, you need to define two subnets (referred to in the UI as the public subnet and the private subnet). The VNet and subnet IP ranges that you use for Azure Databricks defines the maximum number of cluster nodes that you can use at one time. Choose these values carefully.
    • To implement front-end Private Link, back-end Private Link, or both, your workspace VNet needs a third subnet that contains the Private Link endpoint and its IP address range must not overlap with the range of your other workspace subnets. This article refers to this third subnet as the private endpoint subnet. Examples and screenshots assume the subnet name private-link. This can be as small as CIDR range /27. Do not define any NSG rules for a subnet that contains private endpoints.
    • If you use the UI to create objects, you need to create the network and subnets manually before creating the Azure Databricks workspace. If you want to use a template, the template that Azure Databricks provides creates a VNet and appropriate subnets for you, including the two regular subnets plus another for private endpoints.
  • (For front-end Private Link) For users to access the workspace from your on-premise network, you must add private connectivity from that network to your Azure network. Add this connectivity before configuring Private Link. Typically you would create or use an existing transit VNet, sometimes called a bastion VNet or hub VNet. This VNet must be reachable from the on-premise user environment using Expressroute or a VPN gateway connection.

    For front-end Private Link, Databricks recommends creating a separate VNet for your connectivity to the control plane, rather than sharing the workspace VNet. Note that the transit VNet and its subnet can be in the same region, zone, and resource group as your workspace VNet and its subnets, but they do not have to match.

    If you implement both back-end and front-end connections, for maximum network security, Databricks recommends you use a separate private endpoint for your front-end connection from a separate transit VNet. Databricks recommends that you create a resource group for the separate transit VNet and use a different private DNS zone for that private endpoint. However, sharing one VNet for both back-end and front-end Private Link connections is supported if you prefer to simplify your network architecture. If you use two separate private endpoints, you cannot share the DNS zone.

  • (For front-end Private Link) To test your new workspace connectivity, Databricks recommends testing connectivity from the transit VNet. Depending on your network location during your development, you may need to create a VM in the transit VNet and connect to it. See Step 5: Test authentication to your workspace.

Azure user permissions

  • As Azure user, you have read/write permissions sufficient to:

    • Provision a new Azure Databricks workspace.
    • Create Azure Private Link endpoints in your workspace VNet and also (for front-end usage) your transit VNet.

    If the user who created the private endpoint for the transit VNet does not have owner/contributor permissions for the workspace, then a separate user with owner/contributor permissions for the workspace must manually approve the private endpoint creation request. See Check for pending approval or approve pending private endpoints

Step 1: Create resource groups

  1. In the Azure portal, go to the resource groups blade.

  2. Click Create to create a resource group.

    Repeat this step two times to create two workspace resource groups. Name each resource group for its purpose:

    • A resource group for your workspace
    • Create a separate resource group for your Private Link resources such as your Private Link endpoint and zone. The region does not need to match the workspace region. Use the Azure region that matches the region that you plan to use for your transit VNet. If the workspace region and the transit VNet region do not match, there are additional costs to transfer data between regions.

Step 2: Create or prepare the VNet your workspace will use

You may already have a VNet that you will use, or you can create a new VNet specifically for Azure Databricks.

For the requirements for IP ranges of the VNet and the two required subnets for the workspace, see the article Deploy Azure Databricks in your Azure virtual network (VNet injection).

The VNet and subnet IP ranges that you use for Azure Databricks defines the maximum number of cluster nodes that you can use at one time. Choose these values carefully to match your own organization’s network requirements and maximum cluster nodes that you expect to use at one time with Azure Databricks. See Address space and maximum cluster nodes

For example, you could create a VNet with these values:

  • IP range: First remove the default IP range, and then add IP range 10.28.0.0/23.
  • Create subnet public-subnet with range 10.28.0.0/25.
  • Create subnet private-subnet with range 10.28.0.128/25.
  • Create subnet private-link with range 10.28.1.0/27.

Step 3: Provision an Azure Databricks workspace and private endpoints

Deploy a new Azure Databricks workspace with the following settings:

To deploy a workspace with these settings, you have several options, including a user interface in the Azure portal, a custom template (which you can apply in the UI, with Azure CLI, or PowerShell), or Terraform. To use a custom template to create a Private Link enabled workspace, use this template.

No matter which approach you choose, carefully set these three values as you create your new workspace:

  • Public network access (for front-end Private Link) (in the template as publicNetworkAccess): Controls your settings for the front-end use case of Private Link.
    • If you set this to Enabled, users and REST API clients on the public internet can access Azure Databricks, although you can limit access to specific IP ranges from approved source networks using IP access lists.
    • If you set this to Disabled, no user access is permitted from the public internet. If set to Disabled, the front-end connection can be accessed only using PrivateLink connectivity and not from the public internet. IP access lists are not effective on Private Link connections.
    • If you wish to use only back-end Private Link (no front-end Private Link), you must set public network access to Enabled.
  • Required NSG rules (for back-end Private Link) (in the template as requiredNsgRules): Possible values:
    • All Rules: (in the template as AllRules): This setting indicates that your workspace data plane needs a network security group that includes Azure Databricks rules that allow-list connections on the public internet from the data plane to the control plane. If you are not using back-end Private Link, use this setting.
    • No Azure Databricks Rules (in the template as NoAzureDatabricksRules): Use this setting if you are using back-end Private Link, which means that your workspace data plane does not need network security group rules to connect to the Azure Databricks control plane. If you are using back-end Private Link, use this setting.
  • Enable secure cluster cluster connectivity (No-Public-IP/NPIP) (in the template as enableNoPublicIp): Always set to Yes (true), which enables secure cluster connectivity.

Important

These Private Link configuration settings must be defined as part of workspace creation. You cannot add or update these values later on a running workspace. Choose carefully what settings to use before deployment.

The combination of the settings Public network access (in the template, publicNetworkAccess) and Required NSG rules (in the template, requiredNsgRules) define what types of Private Link are supported.

The following table shows the supported scenarios for the two main Private Link use cases, which are front-end and back-end.

Scenario Set public network access to this value Set Required NSG rules to this value Create these endpoints
No Private Link for front-end or back-end Enabled All Rules n/a
Recommended configuration: Both front-end and back-end Private Link. Front-end connectivity is locked down to require Private Link. Disabled NoAzureDatabricksRules One for back-end (required). One for front-end (required). Additionally, one browser auth private endpoint per region.
Both front-end and back-end Private Link. Hybrid front-end connectivity allows Private Link or public internet, perhaps using IP access lists. Enabled NoAzureDatabricksRules One for back-end (required). One endpoint for front-end (optional). Additionally, one browser auth private endpoint per region.
Front-end only Private Link. Front-end connectivity is locked down to require Private Link (public network access is disabled). No Private Link for back-end. This is an unsupported scenario. This is an unsupported scenario. This is an unsupported scenario.
Front-end only Private Link. Hybrid front-end connectivity allows Private Link or public internet, perhaps using IP access lists. No Private Link for back-end. Enabled All Rules One endpoint for front-end (optional). Additionally, one browser auth private endpoint per region.
Back-end only Private Link. Front-end uses public internet, perhaps using IP access lists. No Private Link for front-end. Enabled NoAzureDatabricksRules One endpoint for back-end (required).

The one unsupported combination of values for public network access and Required NSG rules represents the unsupported configuration of front-end only Private Link. The unsupported combination is Public network access set to Disabled and Required NSG rules set to All Rules.

In all cases, you must set these workspace configuration settings:

  • Set Pricing tier to Premium (in a template, this value is premium)
  • Set Enable No Public Ip (secure cluster connectivity) to Yes (in a template, this value is true).
  • Set Networking > Deploy Azure Databricks workspace in your own Virtual Network (Vnet) to Yes (in a template, this value is true)

Note

In general, you need to enable Private Link as part of creation of new workspace. However, if you have an existing workspace that never had Private Link front-end or back-end access, or if you used the default values for Public network access (Enabled) and Required NSG rules (All Rules), you can choose to add front-end Private Link later. However, Public network access will remain enabled so only some of the configuration options are available to you.

There are two ways you can create the workspace:

Create the workspace and private endpoints in the Azure portal UI

The Azure portal automatically includes the two Private Link fields (Public network access and Required NSG rules) when creating a new Azure Databricks workspace.

  1. To create the workspace with your own VNet (VNet injection). To configure and size your subnets, follow the workspace procedure in Deploy Azure Databricks in your Azure virtual network (VNet injection) but do not yet press Create.

    Set the following fields:

    1. Set Pricing Tier to premium or else you will not see the Private Link fields in the UI.
    2. Set Networking > Deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP) to Yes.
    3. Set Networking > Deploy Azure Databricks workspace in your own Virtual Network (Vnet) to Yes.
    4. Set the subnets according to the VNet that you created in a previous step. For details, see the article for VNet injection.
    5. Set Private Link workspace fields Public Network Access and Required Nsg Rules according to the scenarios table in Step 3: Provision an Azure Databricks workspace and private endpoints.

    The following screenshot shows the four most important fields for Private Link connectivity.

    Azure portal UI for workspace

  2. Create a private endpoint for back-end connectivity:

    1. Look for the Private endpoints section below the fields shown in the previous screenshot. If you do not see them, you probably did not set the Pricing Tier to Premium.

      PL private endpoints empty

    2. Click + Add.

      Azure portal shows the Create private endpoint blade within the create workspace.

      Create private endpoint blade

      When you create the private endpoint from within the workspace some Azure fields for this object type are not shown because they are auto-populated and non-editable. Some fields are visible but do not need to be edited:

      • The Azure Databricks sub-resource field is visible and automatically populated with the value databricks_ui_api. That sub-resource value represents the current Azure Databricks control plane for your workspace. This sub-resource name value is used for private endpoints for both back-end and front-end connectivity.

      • After you set your resource group, VNet, and subnet, the Private DNS Zone is automatically populated with a value if you use the Azure-created built-in DNS rather than a custom DNS.

        Important

        Azure might not automatically choose the Private DNS zone that you want to use. Review the value for the Private DNS Zone field and modify it as needed.

    3. Set the Location to match your workspace region. Note that for back-end private endpoint region and workspace region must match, even though for front-end private endpoint connections, the regions do not need to match.

    4. Set the Virtual network to your workspace VNet.

    5. Set the subnet to Private Link specific subnet in your workspace. For related information, see network requirements.

    6. For typical use with the built-in Azure DNS, set Integrate with private DNS zone to Yes. The rest of these instructions assume you chose Yes.

      If your organization maintains its own custom DNS, you may want to set this to No, but review this Microsoft article on DNS configuration before proceeding. Contact your Azure Databricks representative if you have questions.

    7. Click OK to create the private endpoint and return to the workspace creation blade.

    8. In the Public Preview release, only one private endpoint can be created directly from within the workspace creation flow. To create a separate front-end private endpoint from your transit VNet, you need to create an additional private endpoint but you need to do that after the workspace is deployed.

      To finalize creation of the workspace, click Review + create and then Create.

      Wait until the workspace is deployed, and click on Go to resource. This is the Azure portal object for your Azure Databricks workspace. Consider pinning this resource to your Azure dashboard for easy access to it.

  3. Create an additional front-end private endpoint to connect your transit VNet to the Azure Databricks control plane:

    1. From the workspace object in the Azure portal, click Networking.
    2. Click the Private endpoint connections tab.
    3. Click + Private endpoint.
    4. Set the resource group to your resource group for your front-end connection.
    5. For the Region, a front-end private endpoint must be in the same region as your transit VNet, but can be in a different region than your workspace or control plane.

Create the workspace by using a custom template and optionally add front-end private endpoints

If you do not want to use the standard Azure portal UI to create the workspace, you can use a template to deploy your workspace. You can use the template with:

The all-in-one deployment ARM template creates the following resources:

  • Network security groups

  • Resource groups

  • VNet including subnets for the workspace (the standard two subnets) and Private Link (an additional subnet)

  • Azure Databricks workspace

  • The back-end Private Link endpoint

    Note

    The template does not create a front-end endpoint from your transit VNet. After creation of the workspace, you can add that endpoint manually.

  1. You can deploy the template directly from the main page for the template

  2. To deploy it directly, click Deploy on Azure. To view the source, click Browse on GitHub.

    In both cases, set the following parameter values for the template:

    • Set pricingTier to premium. If you leave this as standard, the Azure portal hides the configuration fields that are specific to Private Link.
    • Set enableNoPublicIp to true
    • Set publicNetworkAccess and requiredNsgRules according to the table in Step 3: Provision an Azure Databricks workspace and private endpoints
    • Set the networkSecurityGroup to the ID for your workspace NSG.
  3. Wait for the workspace to deploy.

  4. Navigate to the new **Azure Databricks Service resource that represents your workspace. This is the Azure portal object for your Azure Databricks workspace. Consider pinning this resource to your Azure dashboard for easy access to it.

  5. (Front-end Private Link only) Create the front-end connection to your transit VNet:

    1. In the left navigation, click Settings > Networking.
    2. Click the Private endpoint connections tab.
    3. Click + Private endpoint.
    4. Set the resource group to your resource group for your front-end connection.
    5. For the Region, your front-end private endpoint must be in the same region as your transit VNet, but can be in a different region than your workspace or control plane.

Step 4: Configure DNS to support SSO authentication flow (required for UI access)

User authentication to the Azure Databricks web application uses OAuth as part of the Azure Active Directory SSO implementation. During authentication, the user browser connects to the Azure Databricks control plane. In addition, the OAuth flow requires a network callback redirect from Azure Active Directory. If you configured front-end Private Link, without additional configuration the SSO network redirect fails. This means that users in the transit VNet would be unable to authenticate to Azure Databricks. Note that this issue applies to user login to the web application UI over a front-end connection but does not apply to REST API connections because REST API authentication doesn’t use SSO callbacks.

To properly support authentication and single sign-on (SSO) authentication callbacks from Azure Active Directory, you must create what is called a browser authentication private endpoint, which is a private endpoint with the sub-resource called browser_authentication. Creating this causes Azure Databricks to automatically configure the DNS records for the callback from Azure Active Directory during SSO login. The DNS changes are made by default in the private DNS zone that is associated with the workspace VNet.

For an organization with multiple workspaces, it is important to understand that a properly configured network configuration is exactly one browser authentication private endpoint for each Azure Databricks region. It configures private web authentication for all workspaces in the region with Private Link front-end connectivity. For example, if you have one transit VNet region and 10 production workspaces in the us-west1 region, you will have one browser authentication private endpoint in your transit VNet and it supports all 10 production workspaces.

Important

  • If someone deletes the workspace that hosts the browser authentication private endpoint for that region, it disrupts user web authentication for any other workspaces in that region that relied on that browser authentication private endpoint and related DNS configuration for SSO callbacks.
  • To reduce risks from workspace deletion and encourage standard workspace configuration for your production workspaces, Databricks strongly recommends that you create a private web auth workspace for each region.

A private web auth workspace is a workspace that you create in the same region as your Azure Databricks workspaces, and its only purpose is hosting the browser authentication private endpoint connection from a specific transit VNet to your actual production Azure Databricks workspaces in that region. In all other ways, the private web auth workspace is not used for anything, for example do not use it for running jobs or other workloads. It does not need any actual user data or incoming network connectivity other than the browser authentication private endpoint. You can configure it to have no user access. By setting the workspace setting Public network access to Disabled and do not create any front-end private endpoints to the workspace, users do not have access to user login to the workspace.

Note

To visualize how the private web auth workspace works with other objects for Private Link connectivity, see the diagram earlier in this article.

The private web auth workspace acts as a connection hub between the transit VNet and your production workspaces. It is only used to host the DNS records that allow SSO callbacks to work successfully. After login to your regular workspaces, after login is complete, the private web auth workspace is unused until the next login.

Whether or not you choose to create a private web auth workspace, you must choose one workspace in the region that will host the web authentication private endpoint destination. From the perspective of the Azure portal, the workspace contains the web auth private endpoint. At run time the actual network access is from your transit VNet to Azure Active Directory. After successful login using Azure AD, the user’s web browser is redirected to the correct control plane instance.

Databricks strongly recommends the private web auth workspace configuration if you have multiple workspaces that share a private DNS configuration. You can choose to omit the private web auth workspace if you have only one workspace or are confident that you will not need to delete any workspace. In that case, choose one of your production workspaces to host the browser authentication private endpoint, but be aware of the risks that eventual deletion of that workspace will prevent user authentication for other workspaces in the region with front-end Private Link support.

  1. Recommended but optional: create a private web auth workspace to host the web authentication service. Create one private web auth workspace for each region into which you have deployed Azure Databricks workspaces.

    • You can use the Azure portal, Azure CLI, Powershell, or Terraform to create a new Azure Databricks workspace.
    • Set the workspace name to WEB_AUTH_DO_NOT_DELETE_<region> to ensure it is not deleted.
    • Set Required NSG Rules (requiredNsgRules) to the value NoAzureDatabricksRules.
    • Set Secure cluster connectivity (NPIP) (disablePublicIp) to Enabled.
    • Create the workspace in the same region as your other production workspaces.
    • Use VNet injection. Create or use a VNet separate from the VNet that you use for your main workspace VNet.
    • It does not need to have the same requirements as listed earlier in this article for your regular workspace.
    • Set Public network access (publicNetworkAccess) to Disabled.
    • Do not put any Azure Databricks workload on this workspace.
    • Do not add any private endpoints other than the browser authentication private endpoint. For example, do not create any private endpoint with the databricks_ui_api sub-resource, which would enable front-end or back-end connections to the workspace, which is not necessary.

    For details of deploying a workspace using VNet injection, see Deploy Azure Databricks in your Azure virtual network (VNet injection)

    To create the workspace you can use this all-in-one ARM template with private endpoint support and follow the requirements listed above for the workspace configuration.

  2. In Azure portal, navigate to the Azure Databricks Service instance that represents your workspace.

    • If you are using a private web auth workspace, go to the Azure Databricks Service instance object for the private web auth workspace.
    • If you are not using a private web auth workspace, choose one workspace that will host the web auth private endpoint. Remember that deletion of that workspace will delete DNS records that are required for all your other workspaces in that region that use Private Link front-end connections.
  3. Go to Settings > Networking > Private endpoint connections.

  4. Click the + Add button to create a private endpoint for this workspace.

  5. The Azure portal shows the Create private endpoint blade within the create workspace flow.

    create private endpoint

  6. In the Resource step, set the Target sub-resource field to browser_authentication.

    Note that the Resource type and Resource fields automatically reference the Azure Databricks Service workspace instance that you are editing.

    browser auth sub-resource

  7. In the Virtual Network step:

    endpoint VNet step

    • Set the virtual network to your transit VNet.
    • Set the subnet to the Private Link specific subnet in your transit VNet. If you have not created that subnet yet, do that now in another browser window. For related information, see network requirements.
  8. In the DNS step:

    endpoint DNS step

    • For typical use with the built-in Azure DNS, set Integrate with private DNS zone to Yes.

      If your organization maintains its own custom DNS, you could set Integrate with private DNS zone to No, but read this Microsoft article on DNS configuration before proceeding. Contact your Azure Databricks representative if you have questions.

      The rest of the instructions in this article assume you chose Yes.

    • Verify that the resource group is set to the right resource group. It may have been pre-populated to the correct resource group, but it is not guaranteed to do so. You must choose the same VNet as the front-end private endpoint (not the browser authentication private endpoint).

      Important

      This is a common step for misconfiguration, so do this step carefully.

  9. Click OK to create the private endpoint.

  10. If you integrate with built-in Azure DNS, you can now confirm that your DNS was automatically configured by the browser authentication private endpoint that you created. For example, if you look inside your private DNS zone, you will see one or more new A records with names that end in .pl-auth. These are records that represent the SSO callbacks for each control plane instance in the region. If there is more than one Azure Databricks control plane instance in that region, there will be more than one A record.

    dns updates

Custom DNS

If you use custom DNS, you must ensure that appropriate DNS configuration is configured to support SSO authentication callbacks. For guidance, contact your Databricks representative.

If you are using a front-end private endpoint and users access the Azure Databricks workspace from a transit VNet for which you have enabled custom DNS, you must enable the private endpoint IP address for the workspace to be accessible using the workspace URL.

You may need to configure conditional forwarding to Azure or create a DNS A record for the workspace URL in custom DNS (on-premises or internal DNS). For detailed instructions about how to enable access to Private Link enabled services, see Azure Private Endpoint DNS configuration.

For example, if you directly map the resource URLs to the front-end private endpoint IP addresses in your internal DNS, you need two entries:

  • One entry maps the per-workspace URL (adb-1111111111111.15.azuredatabricks.net) to the front-end private endpoint IP address.
  • One entry maps the Azure Active Directory OAuth flow reply URL (for example westus.pl-auth.azuredatabricks.net) to the front-end private endpoint IP address.

You need to configure the OAuth CNAME record for only one per control plane instance across all workspaces for each VNet because there is no workspace level information shared. As an alternative, you could configure this traffic to egress via the public network, if sharing a single user-to-workspace private endpoint IP address for all business units would be a problem due to common DNS.

Once these configurations are prepared, you should be able to access the Azure Databricks workspace and start clusters for your workloads.

Step 5: Test authentication to your workspace

You must test authentication to your new workspace. For the initial authentication attempt, launch the workspace from within the Azure portal. On the workspace object, there is a button Launch Workspace, which is important. When you click it, Azure Databricks attempts to log in to the workspace and provision your initial admin user account. It is important to test authentication to ensure that your workspace is working correctly.

If you did not configure front-end Private Link, click the button Launch Workspace from any network location.

If you configured front-end Private Link, it is especially important that you test authentication from your transit VNet. Remember that if you have chosen to disallow public network connections, authentication tests can only be performed within the transit VNet or from a VNet that is routable to it. Even if you did not disallow public network connections, it’s still important to test the Private Link connection from your transit VNet.

  1. Confirm network routability to your transit VNet:

    • If you enabled front-end Private Link, you need to test that authentication happens successfully from your transit VNet. If you’ve already used ExpressRoute or VPN to connect your on-premise network to your transit VNet, and if as a user you are at a routable location that can connect to the transit VNet, then you can test Azure Databricks authentication by logging in from your current network.
    • If you are not at a network location that is routable to your transit VNet, then you can test connectivity by creating a virtual machine in your transit VNet.
      1. Go to the Virtual Machine blade in the Azure portal.
      2. Create a Windows 10 virtual machine in the transit VNet. You might want to create it in the same resource group as your transit VNet to make it easy to find and later destroy this temporary VM for testing.
      3. Connect to it with an RDP client such as Microsoft Remote Desktop. Use the VM’s public IP address (which you can get from the Azure portal) and credentials that you used to create the VM.
  2. From within the VM (if you created a VM in the transit VNet) or from your current location (if your location is routable to the transit VNet), use the web browser to connect to the Azure portal. If your VM doesn’t have access to the public internet, you can test private link access via directly accessing the per-workspace URL for your workspace.

  3. In the Azure portal, find the Azure Databricks workspace object.

  4. Click Launch Workspace to launch a window tab that logs you into Azure Databricks using your user ID that you used to log in to the Azure portal.

    Confirm that logging in is successful.

Authentication troubleshooting

Error: If you see a message “Configured privacy settings disallow access for workspace <your-workspace-id> over your current network. Please contact your administrator for more information.”

This error probably means:

  • You are connecting to the workspace over the public internet (not from a Private Link connection).
  • You have configured the workspace to not support public network access.

Review the earlier steps in this section.

Error: “Browser failure with error code DNS_PROBE_FINISHED_NXDOMAIN

This error means that user SSO logon to the Azure Databricks web application failed because it could not find the appropriate DNS configuration for the Azure Databricks control plane instance in the target region. The DNS record that points to the <control-plane-instance-short-name>.pl-auth name was not found. You may have misconfigured the web authentication private endpoint. Carefully re-review the section Step 4: Configure DNS to support SSO authentication flow (required for UI access). If you have questions, contact your Azure Databricks representative.

If you added a back-end Private Link connection, it’s important to test that it is working correctly. Simply logging into the Azure Databricks web application does not test the back-end connection.

  1. If you are not already logged into your Azure Databricks workspace, log in again using your workspace URL or from the Launch Workspace button in the Azure Databricks Service instance in the Azure portal.

  2. In the left nav, click Compute

  3. Click Create Cluster, type a cluster name, and click Create Cluster. For more information about creating clusters, see Configure clusters.

    Wait until the cluster appears to be started successfully. It may take several minutes. Restart the page to get the latest status.

How to delete an Azure workspace that has private endpoints

Important

If you are using the recommended but optional deployment style that uses a private web auth workspace, it’s important that you never delete the workspace or the browser auth private endpoint that is associated with the workspace. See Step 4: Configure DNS to support SSO authentication flow (required for UI access).

Azure blocks the deletion of a Azure Databricks workspace if it has any private endpoints.

Important

You must delete the private endpoints before attempting to delete the Azure Databricks workspace.

  1. In the Azure portal, open the Azure Databricks Service instance that represents your workspace.
  2. In the left nav, click Settings > Networking.
  3. Click the Private endpoint connections tab.
  4. If you are not using a private web auth workspace, check if your organization may be using this workspace as an OAuth CNAME link and it may be shared with other workspaces that use the same control plane instance. If so, before you delete any private endpoints that may rely on the CNAME from this workspace, configure the other workspace’s network objects to ensure that the CNAME still points to a valid zone A record from another workspace. See Step 4: Configure DNS to support SSO authentication flow (required for UI access).
  5. For each private endpoint, select the row and click on the Remove icon. Click Yes to confirm removal.

When you are done, you can delete the workspace from Azure portal.

Check for pending approval or approve pending private endpoints

If the Azure user who created the private endpoint for the transit VNet does not have owner/contributor permissions for the workspace, then a separate user with owner/contributor permissions for the workspace must manually approve the private endpoint creation request before it is enabled.

Connection states include:

  • Approved: Endpoint is approved and no further action is necessary.
  • Pending: Endpoint requires approval from an admin with owner/contributor permissions for the workspace.
  • Disconnected: The endpoint because a related object for this connection was deleted.
  • Rejected: The endpoint was rejected by an admin.

To check connection state:

  1. In the Azure portal, navigate to your workspace that contains one or more private endpoints that you have recently created.

  2. Click Networking.

  3. Click the Private endpoint connections tab.

  4. In the list of endpoints, look at the Connection state column.

    • If they all have the value connection state value Approved, there is no action necessary to approve the private endpoint.
    • If the value of any are Pending, they require approval from someone with owner/contributor permissions for the workspace.
  5. If you have owner/contributor permissions for the workspace:

    1. Select one or more endpoint rows that are pending.

    2. If you approve of the connection, click the Approve button.

      If you disapprove of the connection, click the Reject button.

    If you do not have owner/contributor permissions for the workspace, contact your workspace administrator to approve the connection.