Data encryption with Azure Machine Learning
Azure Machine Learning uses a variety of Azure data storage services and compute resources when training models and performing inference. Each of these has their own story on how they provide encryption for data at rest and in transit. In this article, learn about each one and which is best for your scenario.
For production grade encryption during training, Microsoft recommends using Azure Machine Learning compute cluster. For production grade encryption during inference, Microsoft recommends using Azure Kubernetes Service.
Azure Machine Learning compute instance is a dev/test environment. When using it, we recommend that you store your files, such as notebooks and scripts, in a file share. Your data should be stored in a datastore.
Encryption at rest
If your workspace contains sensitive data we recommend setting the hbi_workspace flag while creating your workspace. The
hbi_workspace flag can only be set when a workspace is created. It cannot be changed for an existing workspace.
hbi_workspace flag controls the amount of data Microsoft collects for diagnostic purposes and enables additional encryption in Microsoft-managed environments. In addition, it enables the following actions:
- Starts encrypting the local scratch disk in your Azure Machine Learning compute cluster provided you have not created any previous clusters in that subscription. Else, you need to raise a support ticket to enable encryption of the scratch disk of your compute clusters
- Cleans up your local scratch disk between runs
- Securely passes credentials for your storage account, container registry, and SSH account from the execution layer to your compute clusters using your key vault
When this flag is set to True, one possible impact is increased difficulty troubleshooting issues. This could happen because some telemetry isn't sent to Microsoft and there is less visibility into success rates or problem types, and therefore may not be able to react as proactively when this flag is True.
hbi_workspace flag does not impact encryption in transit, only encryption at rest.
Azure Blob storage
Azure Machine Learning stores snapshots, output, and logs in the Azure Blob storage account that's tied to the Azure Machine Learning workspace and your subscription. All the data stored in Azure Blob storage is encrypted at rest with Microsoft-managed keys.
For information on how to use your own keys for data stored in Azure Blob storage, see Azure Storage encryption with customer-managed keys in Azure Key Vault.
Training data is typically also stored in Azure Blob storage so that it's accessible to training compute targets. This storage isn't managed by Azure Machine Learning but mounted to compute targets as a remote file system.
If you need to rotate or revoke your key, you can do so at any time. When rotating a key, the storage account will start using the new key (latest version) to encrypt data at rest. When revoking (disabling) a key, the storage account takes care of failing requests. It usually takes an hour for the rotation or revocation to be effective.
For information on regenerating the access keys, see Regenerate storage access keys.
Azure Cosmos DB
Azure Machine Learning stores metadata in an Azure Cosmos DB instance. This instance is associated with a Microsoft subscription managed by Azure Machine Learning. All the data stored in Azure Cosmos DB is encrypted at rest with Microsoft-managed keys.
To use your own (customer-managed) keys to encrypt the Azure Cosmos DB instance, you can create a dedicated Cosmos DB instance for use with your workspace. We recommend this approach if you want to store your data, such as run history information, outside of the multi-tenant Cosmos DB instance hosted in our Microsoft subscription.
To enable provisioning a Cosmos DB instance in your subscription with customer-managed keys, perform the following actions:
Register the Microsoft.MachineLearning and Microsoft.DocumentDB resource providers in your subscription, if not done already.
Use the following parameters when creating the Azure Machine Learning workspace. Both parameters are mandatory and supported in SDK, Azure CLI, REST APIs, and Resource Manager templates.
cmk_keyvault: This parameter is the resource ID of the key vault in your subscription. This key vault needs to be in the same region and subscription that you will use for the Azure Machine Learning workspace.
resource_cmk_uri: This parameter is the full resource URI of the customer managed key in your key vault, including the version information for the key.
Enabling soft delete and purge protection on the CMK key vault instance is required before creating an encrypted machine learning workspace to protect against accidental data loss in case of vault deletion.
This key vault instance can be different than the key vault that is created by Azure Machine Learning when you provision the workspace. If you want to use the same key vault instance for the workspace, pass the same key vault while provisioning the workspace by using the key_vault parameter.
The Cosmos DB instance is created in a Microsoft-managed resource group in your subscription. The following services are also created in this resource group, and are used by the customer-managed key configuration:
- Azure Storage Account
- Azure Search
Since these services are created in your Azure subscription, it means that you are charged for these service instances. If your subscription does not have enough quota for the Azure Cosmos DB service, a failure will occur. For more information on quotas, see Azure Cosmos DB service quotas
The managed resource group is named in the format
<AML Workspace Resource Group Name><GUID>. If your Azure Machine Learning workspace uses a private endpoint, a virtual network is also created in this resource group. This VNet is used to secure communication between the services in this resource group and your Azure Machine Learning workspace.
- Do not delete the resource group that contains this Cosmos DB instance, or any of the resources automatically created in this group. If you need to delete the resource group, Cosmos DB instance, etc., you must delete the Azure Machine Learning workspace that uses it. The resource group, Cosmos DB instance, and other automatically created resources are deleted when the associated workspace is deleted.
- The Request Units used by this Cosmos DB account automatically scale as needed. The minimum RU is 1200. The maximum RU is 12000.
- You cannot provide your own VNet for use with the Cosmos DB instance that is created. You also cannot modify the virtual network. For example, you cannot change the IP address range that it uses.
To estimate the additional cost of the Azure Cosmos DB instance, use the Azure pricing calculator.
If you need to rotate or revoke your key, you can do so at any time. When rotating a key, Cosmos DB will start using the new key (latest version) to encrypt data at rest. When revoking (disabling) a key, Cosmos DB takes care of failing requests. It usually takes an hour for the rotation or revocation to be effective.
For more information on customer-managed keys with Cosmos DB, see Configure customer-managed keys for your Azure Cosmos DB account.
Azure Container Registry
All container images in your registry (Azure Container Registry) are encrypted at rest. Azure automatically encrypts an image before storing it and decrypts it when Azure Machine Learning pulls the image.
To use your own (customer-managed) keys to encrypt your Azure Container Registry, you need to create your own ACR and attach it while provisioning the workspace or encrypt the default instance that gets created at the time of workspace provisioning.
Azure Machine Learning requires the admin account be enabled on your Azure Container Registry. By default, this setting is disabled when you create a container registry. For information on enabling the admin account, see Admin account.
Once an Azure Container Registry has been created for a workspace, do not delete it. Doing so will break your Azure Machine Learning workspace.
For an example of creating a workspace using an existing Azure Container Registry, see the following articles:
- Create a workspace for Azure Machine Learning with Azure CLI.
- Create a workspace with Python SDK.
- Use an Azure Resource Manager template to create a workspace for Azure Machine Learning
Azure Container Instance
You may encrypt a deployed Azure Container Instance (ACI) resource using customer-managed keys. The customer-managed key used for ACI can be stored in the Azure Key Vault for your workspace. For information on generating a key, see Encrypt data with a customer-managed key.
To use the key when deploying a model to Azure Container Instance, create a new deployment configuration using
AciWebservice.deploy_configuration(). Provide the key information using the following parameters:
cmk_vault_base_url: The URL of the key vault that contains the key.
cmk_key_name: The name of the key.
cmk_key_version: The version of the key.
For more information on creating and using a deployment configuration, see the following articles:
- AciWebservice.deploy_configuration() reference
- Where and how to deploy
- Deploy a model to Azure Container Instances
For more information on using a customer-managed key with ACI, see Encrypt data with a customer-managed key.
Azure Kubernetes Service
You may encrypt a deployed Azure Kubernetes Service resource using customer-managed keys at any time. For more information, see Bring your own keys with Azure Kubernetes Service.
This process allows you to encrypt both the Data and the OS Disk of the deployed virtual machines in the Kubernetes cluster.
This process only works with AKS K8s version 1.17 or higher. Azure Machine Learning added support for AKS 1.17 on Jan 13, 2020.
Machine Learning Compute
Compute cluster The OS disk for each compute node stored in Azure Storage is encrypted with Microsoft-managed keys in Azure Machine Learning storage accounts. This compute target is ephemeral, and clusters are typically scaled down when no runs are queued. The underlying virtual machine is de-provisioned, and the OS disk is deleted. Azure Disk Encryption isn't supported for the OS disk.
Each virtual machine also has a local temporary disk for OS operations. If you want, you can use the disk to stage training data. If the workspace was created with the
hbi_workspace parameter set to
TRUE, the temporary disk is encrypted. This environment is short-lived (only for the duration of your run,) and encryption support is limited to system-managed keys only.
The OS disk for compute instance is encrypted with Microsoft-managed keys in Azure Machine Learning storage accounts. If the workspace was created with the
hbi_workspace parameter set to
TRUE, the local temporary disk on compute instance is encrypted with Microsoft managed keys. Customer managed key encryption is not supported for OS and temp disk.
Azure Databricks can be used in Azure Machine Learning pipelines. By default, the Databricks File System (DBFS) used by Azure Databricks is encrypted using a Microsoft-managed key. To configure Azure Databricks to use customer-managed keys, see Configure customer-managed keys on default (root) DBFS.
When using services such as Automated Machine Learning, Microsoft may generate a transient, pre-processed data for training multiple models. This data is stored in a datastore in your workspace, which allows you to enforce access controls and encryption appropriately.
You may also want to encrypt diagnostic information logged from your deployed endpoint into your Azure Application Insights instance.
Encryption in transit
Azure Machine Learning uses TLS to secure internal communication between various Azure Machine Learning microservices. All Azure Storage access also occurs over a secure channel.
To secure external calls made to the scoring endpoint, Azure Machine Learning uses TLS. For more information, see Use TLS to secure a web service through Azure Machine Learning.
Data collection and handling
Microsoft collected data
Microsoft also recommends not storing sensitive information (such as account key secrets) in environment variables. Environment variables are logged, encrypted, and stored by us. Similarly when naming run_id, avoid including sensitive information such as user names or secret project names. This information may appear in telemetry logs accessible to Microsoft Support engineers.
You may opt out from diagnostic data being collected by setting the
hbi_workspace parameter to
TRUE while provisioning the workspace. This functionality is supported when using the AzureML Python SDK, the Azure CLI, REST APIs, or Azure Resource Manager templates.
Using Azure Key Vault
Azure Machine Learning uses the Azure Key Vault instance associated with the workspace to store credentials of various kinds:
- The associated storage account connection string
- Passwords to Azure Container Repository instances
- Connection strings to data stores
SSH passwords and keys to compute targets like Azure HDInsight and VMs are stored in a separate key vault that's associated with the Microsoft subscription. Azure Machine Learning doesn't store any passwords or keys provided by users. Instead, it generates, authorizes, and stores its own SSH keys to connect to VMs and HDInsight to run the experiments.
Each workspace has an associated system-assigned managed identity that has the same name as the workspace. This managed identity has access to all keys, secrets, and certificates in the key vault.