How to create a secure workspace

In this article, learn how to create and connect to a secure Azure Machine Learning workspace. A secure workspace uses Azure Virtual Network to create a security boundary around resources used by Azure Machine Learning.

In this tutorial, you accomplish the following tasks:

  • Create an Azure Virtual Network (VNet) to secure communications between services in the virtual network.
  • Create an Azure Storage Account (blob and file) behind the VNet. This service is used as default storage for the workspace.
  • Create an Azure Key Vault behind the VNet. This service is used to store secrets used by the workspace. For example, the security information needed to access the storage account.
  • Create an Azure Container Registry (ACR). This service is used as a repository for Docker images. Docker images provide the compute environments needed when training a machine learning model or deploying a trained model as an endpoint.
  • Create an Azure Machine Learning workspace.
  • Create a jump box. A jump box is an Azure Virtual Machine that is behind the VNet. Since the VNet restricts access from the public internet, the jump box is used as a way to connect to resources behind the VNet.
  • Configure Azure Machine Learning studio to work behind a VNet. The studio provides a web interface for Azure Machine Learning.
  • Create an Azure Machine Learning compute cluster. A compute cluster is used when training machine learning models in the cloud. In configurations where Azure Container Registry is behind the VNet, it is also used to build Docker images.
  • Connect to the jump box and use the Azure Machine Learning studio.

Tip

If you're looking for a template (Microsoft Bicep or Hashicorp Terraform) that demonstrates how to create a secure workspace, see Tutorial - Create a secure workspace using a template.

Prerequisites

  • Familiarity with Azure Virtual Networks and IP networking. If you are not familiar, try the Fundamentals of computer networking module.
  • While most of the steps in this article use the Azure portal or the Azure Machine Learning studio, some steps use the Azure CLI extension for Machine Learning v2.

Limitations

The steps in this article put Azure Container Registry behind the VNet. In this configuration, you can't deploy models to Azure Container Instances inside the VNet. For more information, see Secure the inference environment.

Create a virtual network

To create a virtual network, use the following steps:

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Virtual Network in the search field. Select the Virtual Network entry, and then select Create.

    The create resource UI search

    Virtual network create

  2. From the Basics tab, select the Azure subscription to use for this resource and then select or create a new resource group. Under Instance details, enter a friendly name for your virtual network and select the region to create it in.

    Image of the basic virtual network config

  3. Select IP Addresses tab. The default settings should be similar to the following image:

    Default IP Address screen

    Use the following steps to configure the IP address and configure a subnet for training and scoring resources:

    Tip

    While you can use a single subnet for all Azure ML resources, the steps in this article show how to create two subnets to separate the training & scoring resources.

    The workspace and other dependency services will go into the training subnet. They can still be used by resources in other subnets, such as the scoring subnet.

    1. Look at the default IPv4 address space value. In the screenshot, the value is 172.17.0.0/16. The value may be different for you. While you can use a different value, the rest of the steps in this tutorial are based on the 172.16.0.0/16 value.

      Important

      We do not recommend using an address in the 172.17.0.1/16 range if you plan on using Azure Kubernetes Services for deployment with this cluster. The Docker bridge in Azure Kubernetes Services uses 172.17.0.1/16 as its default. Other ranges may also conflict depending on what you want to connect to the virtual network. For example, if you plan to connect your on premises network to the VNet, and your on-premises network also uses the 172.16.0.0/16 range. Ultimately, it is up to you to plan your network infrastructure.

    2. Select the Default subnet and then select Remove subnet.

      Screenshot of deleting default subnet

    3. To create a subnet to contain the workspace, dependency services, and resources used for training, select + Add subnet and set the subnet name and address range. The following are the values used in this tutorial:

      • Subnet name: Training
      • Subnet address range: 172.16.0.0/24

      Screenshot of Training subnet

      Tip

      If you plan on using a service endpoint to add your Azure Storage Account, Azure Key Vault, and Azure Container Registry to the VNet, select the following under Services:

      • Microsoft.Storage
      • Microsoft.KeyVault
      • Microsoft.ContainerRegistry

      If you plan on using a private endpoint to add these services to the VNet, you do not need to select these entries. The steps in this article use a private endpoint for these services, so you do not need to select them when following these steps.

    4. To create a subnet for compute resources used to score your models, select + Add subnet again, and set the name and address range:

      • Subnet name: Scoring
      • Subnet address range: 172.16.1.0/24

      Screenshot of Scoring subnet

      Tip

      If you plan on using a service endpoint to add your Azure Storage Account, Azure Key Vault, and Azure Container Registry to the VNet, select the following under Services:

      • Microsoft.Storage
      • Microsoft.KeyVault
      • Microsoft.ContainerRegistry

      If you plan on using a private endpoint to add these services to the VNet, you do not need to select these entries. The steps in this article use a private endpoint for these services, so you do not need to select them when following these steps.

  4. Select Security. For BastionHost, select Enable. Azure Bastion provides a secure way to access the VM jump box you will create inside the VNet in a later step. Use the following values for the remaining fields:

    • Bastion name: A unique name for this Bastion instance
    • AzureBastionSubnetAddress space: 172.16.2.0/27
    • Public IP address: Create a new public IP address.

    Leave the other fields at the default values.

    Screenshot of Bastion config

  5. Select Review + create.

    Screenshot showing the review + create button

  6. Verify that the information is correct, and then select Create.

    Screenshot of the review page

Create a storage account

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Storage account. Select the Storage Account entry, and then select Create.

  2. From the Basics tab, select the subscription, resource group, and region you previously used for the virtual network. Enter a unique Storage account name, and set Redundancy to Locally-redundant storage (LRS).

    Image of storage account basic config

  3. From the Networking tab, select Private endpoint and then select + Add private endpoint.

    UI to add the blob private network

  4. On the Create private endpoint form, use the following values:

    • Subscription: The same Azure subscription that contains the previous resources you've created.
    • Resource group: The same Azure resource group that contains the previous resources you've created.
    • Location: The same Azure region that contains the previous resources you've created.
    • Name: A unique name for this private endpoint.
    • Target sub-resource: blob
    • Virtual network: The virtual network you created earlier.
    • Subnet: Training (172.16.0.0/24)
    • Private DNS integration: Yes
    • Private DNS Zone: privatelink.blob.core.windows.net

    Select OK to create the private endpoint.

  5. Select Review + create. Verify that the information is correct, and then select Create.

  6. Once the Storage Account has been created, select Go to resource:

    Go to new storage resource

  7. From the left navigation, select Networking the Private endpoint connections tab, and then select + Private endpoint:

    Note

    While you created a private endpoint for Blob storage in the previous steps, you must also create one for File storage.

    UI for storage account networking

  8. On the Create a private endpoint form, use the same subscription, resource group, and Region that you have used for previous resources. Enter a unique Name.

    UI to add the file private endpoint

  9. Select Next : Resource, and then set Target sub-resource to file.

    Add the subresource of 'file'

  10. Select Next : Configuration, and then use the following values:

    • Virtual network: The network you created previously
    • Subnet: Training
    • Integrate with private DNS zone: Yes
    • Private DNS zone: privatelink.file.core.windows.net

    UI to configure the file private endpoint

  11. Select Review + Create. Verify that the information is correct, and then select Create.

Tip

If you plan to use ParallelRunStep in your pipeline, it is also required to configure private endpoints target queue and table sub-resources. ParallelRunStep uses queue and table under the hood for task scheduling and dispatching.

Create a key vault

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Key Vault. Select the Key Vault entry, and then select Create.

  2. From the Basics tab, select the subscription, resource group, and region you previously used for the virtual network. Enter a unique Key vault name. Leave the other fields at the default value.

    Create a new key vault

  3. From the Networking tab, select Private endpoint and then select + Add.

    Key vault networking

  4. On the Create private endpoint form, use the following values:

    • Subscription: The same Azure subscription that contains the previous resources you've created.
    • Resource group: The same Azure resource group that contains the previous resources you've created.
    • Location: The same Azure region that contains the previous resources you've created.
    • Name: A unique name for this private endpoint.
    • Target sub-resource: Vault
    • Virtual network: The virtual network you created earlier.
    • Subnet: Training (172.16.0.0/24)
    • Private DNS integration: Yes
    • Private DNS Zone: privatelink.vaultcore.azure.net

    Select OK to create the private endpoint.

    Configure a key vault private endpoint

  5. Select Review + create. Verify that the information is correct, and then select Create.

Create a container registry

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Container Registry. Select the Container Registry entry, and then select Create.

  2. From the Basics tab, select the subscription, resource group, and location you previously used for the virtual network. Enter a unique Registry name and set the SKU to Premium.

    Create a container registry

  3. From the Networking tab, select Private endpoint and then select + Add.

    Container registry networking

  4. On the Create private endpoint form, use the following values:

    • Subscription: The same Azure subscription that contains the previous resources you've created.
    • Resource group: The same Azure resource group that contains the previous resources you've created.
    • Location: The same Azure region that contains the previous resources you've created.
    • Name: A unique name for this private endpoint.
    • Target sub-resource: registry
    • Virtual network: The virtual network you created earlier.
    • Subnet: Training (172.16.0.0/24)
    • Private DNS integration: Yes
    • Private DNS Zone: privatelink.azurecr.io

    Select OK to create the private endpoint.

    Configure container registry private endpoint

  5. Select Review + create. Verify that the information is correct, and then select Create.

  6. After the container registry has been created, select Go to resource.

    Select 'go to resource'

  7. From the left of the page, select Access keys, and then enable Admin user. This setting is required when using Azure Container Registry inside a virtual network with Azure Machine Learning.

    Screenshot of admin user toggle

Create a workspace

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Machine Learning. Select the Machine Learning entry, and then select Create.

    {alt-text}

  2. From the Basics tab, select the subscription, resource group, and Region you previously used for the virtual network. Use the follow values for the other fields:

    • Workspace name: A unique name for your workspace.
    • Storage account: Select the storage account you created previously.
    • Key vault: Select the key vault you created previously.
    • Application insights: Use the default value.
    • Container registry: Use the container registry you created previously.

    Basic workspace configuration

  3. From the Networking tab, select Private endpoint and then select + add.

    Workspace networking

  4. On the Create private endpoint form, use the following values:

    • Subscription: The same Azure subscription that contains the previous resources you've created.
    • Resource group: The same Azure resource group that contains the previous resources you've created.
    • Location: The same Azure region that contains the previous resources you've created.
    • Name: A unique name for this private endpoint.
    • Target sub-resource: amlworkspace
    • Virtual network: The virtual network you created earlier.
    • Subnet: Training (172.16.0.0/24)
    • Private DNS integration: Yes
    • Private DNS Zone: Leave the two private DNS zones at the default values of privatelink.api.azureml.ms and privatelink.notebooks.azure.net.

    Select OK to create the private endpoint.

    Screenshot of workspace private network config

  5. Select Review + create. Verify that the information is correct, and then select Create.

  6. Once the workspace has been created, select Go to resource.

  7. From the Settings section on the left, select Private endpoint connections and then select the link in the Private endpoint column:

    Screenshot of workspace private endpoint connections

  8. Once the private endpoint information appears, select DNS configuration from the left of the page. Save the IP address and fully qualified domain name (FQDN) information on this page, as it will be used later.

    screenshot of IP and FQDN entries

Important

There are still some configuration steps needed before you can fully use the workspace. However, these require you to connect to the workspace.

Enable studio

Azure Machine Learning studio is a web-based application that lets you easily manage your workspace. However, it needs some extra configuration before it can be used with resources secured inside a VNet. Use the following steps to enable studio:

  1. When using an Azure Storage Account that has a private endpoint, add the service principal for the workspace as a Reader for the storage private endpoint(s). From the Azure portal, select your storage account and then select Networking. Next, select Private endpoint connections.

    Screenshot of storage private endpoints

  2. For each private endpoint listed, use the following steps:

    1. Select the link in the Private endpoint column.

      Screenshot of endpoints to select

    2. Select Access control (IAM) from the left side.

    3. Select + Add, and then Add role assignment (Preview).

      Access control (IAM) page with Add role assignment menu open.

    4. On the Role tab, select the Reader.

      Add role assignment page with Role tab selected.

    5. On the Members tab, select User, group, or service principal in the Assign access to area and then select + Select members. In the Select members dialog, enter the name as your Azure Machine Learning workspace. Select the service principal for the workspace, and then use the Select button.

    6. On the Review + assign tab, select Review + assign to assign the role.

Connect to the workspace

There are several ways that you can connect to the secured workspace. The steps in this article use a jump box, which is a virtual machine in the VNet. You can connect to it using your web browser and Azure Bastion. The following table lists several other ways that you might connect to the secure workspace:

Method Description
Azure VPN gateway Connects on-premises networks to the VNet over a private connection. Connection is made over the public internet.
ExpressRoute Connects on-premises networks into the cloud over a private connection. Connection is made using a connectivity provider.

Important

When using a VPN gateway or ExpressRoute, you will need to plan how name resolution works between your on-premises resources and those in the VNet. For more information, see Use a custom DNS server.

Create a jump box (VM)

Use the following steps to create a Data Science Virtual Machine for use as a jump box:

  1. In the Azure portal, select the portal menu in the upper left corner. From the menu, select + Create a resource and then enter Data science virtual machine. Select the Data science virtual machine - Windows entry, and then select Create.

  2. From the Basics tab, select the subscription, resource group, and Region you previously used for the virtual network. Provide values for the following fields:

    • Virtual machine name: A unique name for the VM.

    • Username: The username you will use to login to the VM.

    • Password: The password for the username.

    • Security type: Standard.

    • Image: Data Science Virtual Machine - Windows Server 2019 - Gen1.

      Important

      Do not select a Gen2 image.

    You can leave other fields at the default values.

    Image of VM basic configuration

  3. Select Networking, and then select the Virtual network you created earlier. Use the following information to set the remaining fields:

    • Select the Training subnet.
    • Set the Public IP to None.
    • Leave the other fields at the default value.

    Image of VM network configuration

  4. Select Review + create. Verify that the information is correct, and then select Create.

Connect to the jump box

  1. Once the virtual machine has been created, select Go to resource.

  2. From the top of the page, select Connect and then Bastion.

    Image of the connect/bastion UI

  3. Select Use Bastion, and then provide your authentication information for the virtual machine, and a connection will be established in your browser.

    Image of use bastion dialog

Create a compute cluster and compute instance

A compute cluster is used by your training jobs. A compute instance provides a Jupyter Notebook experience on a shared compute resource attached to your workspace.

  1. From an Azure Bastion connection to the jump box, open the Microsoft Edge browser on the remote desktop.

  2. In the remote browser session, go to https://ml.azure.com. When prompted, authenticate using your Azure AD account.

  3. From the Welcome to studio! screen, select the Machine Learning workspace you created earlier and then select Get started.

    Tip

    If your Azure AD account has access to multiple subscriptions or directories, use the Directory and Subscription dropdown to select the one that contains the workspace.

    Screenshot of the select workspace dialog

  4. From studio, select Compute, Compute clusters, and then + New.

    Screenshot of new compute cluster workflow

  5. From the Virtual Machine dialog, select Next to accept the default virtual machine configuration.

    Screenshot of compute cluster vm settings

  6. From the Configure Settings dialog, enter cpu-cluster as the Compute name. Set the Subnet to Training and then select Create to create the cluster.

    Tip

    Compute clusters dynamically scale the nodes in the cluster as needed. We recommend leaving the minimum number of nodes at 0 to reduce costs when the cluster is not in use.

    Screenshot of new compute cluster settings

  7. From studio, select Compute, Compute instance, and then + New.

    Screenshot of new compute instance workflow

  8. From the Virtual Machine dialog, enter a unique Computer name and select Next: Advanced Settings.

    Screenshot of compute instance vm settings

  9. From the Advanced Settings dialog, , set the Subnet to Training, and then select Create.

    Screenshot of compute instance settings

Tip

When you create a compute cluster or compute instance, Azure Machine Learning dynamically adds a Network Security Group (NSG). This NSG contains the following rules, which are specific to compute cluster and compute instance:

  • Allow inbound TCP traffic on ports 29876-29877 from the BatchNodeManagement service tag.
  • Allow inbound TCP traffic on port 44224 from the AzureMachineLearning service tag.

The following screenshot shows an example of these rules:

Screenshot of NSG

For more information on creating a compute cluster and compute cluster, including how to do so with Python and the CLI, see the following articles:

Configure image builds

APPLIES TO: Azure CLI ml extension v1 v2 (preview)

When Azure Container Registry is behind the virtual network, Azure Machine Learning can't use it to directly build Docker images (used for training and deployment). Instead, configure the workspace to use the compute cluster you created earlier. Use the following steps to create a compute cluster and configure the workspace to use it to build images:

  1. Navigate to https://shell.azure.com/ to open the Azure Cloud Shell.

  2. From the Cloud Shell, use the following command to install the 1.0 CLI for Azure Machine Learning:

    az extension add -n ml
    
  3. To update the workspace to use the compute cluster to build Docker images. Replace docs-ml-rg with your resource group. Replace docs-ml-ws with your workspace. Replace cpu-cluster with the compute cluster to use:

    az ml workspace update \
      -n myworkspace \
      -g myresourcegroup \
      -i mycomputecluster
    

    Note

    You can use the same compute cluster to train models and build Docker images for the workspace.

Use the workspace

Important

The steps in this article put Azure Container Registry behind the VNet. In this configuration, you cannot deploy a model to Azure Container Instances inside the VNet. We do not recommend using Azure Container Instances with Azure Machine Learning in a virtual network. For more information, see Secure the inference environment.

At this point, you can use studio to interactively work with notebooks on the compute instance and run training jobs on the compute cluster. For a tutorial on using the compute instance and compute cluster, see run a Python script.

Stop compute instance and jump box

Warning

While it is running (started), the compute instance and jump box will continue charging your subscription. To avoid excess cost, stop them when they are not in use.

The compute cluster dynamically scales between the minimum and maximum node count set when you created it. If you accepted the defaults, the minimum is 0, which effectively turns off the cluster when not in use.

Stop the compute instance

From studio, select Compute, Compute clusters, and then select the compute instance. Finally, select Stop from the top of the page.

Screenshot of stop button for compute instance

Stop the jump box

Once it has been created, select the virtual machine in the Azure portal and then use the Stop button. When you are ready to use it again, use the Start button to start it.

Screenshot of stop button for the VM

You can also configure the jump box to automatically shut down at a specific time. To do so, select Auto-shutdown, Enable, set a time, and then select Save.

Screenshot of auto-shutdown option

Clean up resources

If you plan to continue using the secured workspace and other resources, skip this section.

To delete all resources created in this tutorial, use the following steps:

  1. In the Azure portal, select Resource groups on the far left.

  2. From the list, select the resource group that you created in this tutorial.

  3. Select Delete resource group.

    Screenshot of delete resource group button

  4. Enter the resource group name, then select Delete.

Next steps

Now that you have created a secure workspace and can access studio, learn how to run a Python script using Azure Machine Learning.