Configure customer-managed keys for data encryption in Azure Cognitive Search

Azure Cognitive Search automatically encrypts data at rest with service-managed keys. If more protection is needed, you can supplement default encryption with an additional encryption layer using keys that you create and manage in Azure Key Vault.

This article walks you through the steps of setting up customer-managed key (CMK) or "bring-your-own-key" (BYOK) encryption. Here are some points to keep in mind:

  • CMK encryption is enacted on individual objects. If you require CMK unilaterally across your search service, set an enforcement policy at the service level so that you can be notified if the service falls out of compliance.

  • CMK encryption depends on Azure Key Vault. You can create your own encryption keys and store them in a key vault, or you can use Azure Key Vault APIs to generate encryption keys.

  • CMK encryption occurs when an object is created. You can't encrypt objects that already exist.

CMK encryption support

Objects that can be encrypted include indexes, synonym lists, indexers, data sources, and skillsets. Encryption is computationally expensive to decrypt so only sensitive content is encrypted.

Encryption is performed over the following content:

  • All content within indexes and synonym lists, including descriptions.

  • For indexers, data sources, and skillsets, only those fields that store connection strings, descriptions, keys, and user inputs are encrypted. For example, skillsets have Cognitive Services keys, and some skills accept user inputs, such as custom entities. In both cases, keys and user inputs into skills are encrypted.

Double encryption

Double encryption is an extension of customer-managed key (CMK) encryption. CMK encryption applies to long-term storage that is written to a data disk. The term double encryption refers to the additional encryption of short-term storage (of content written to temporary disks). No configuration is required. When you apply CMK to objects, double encryption is invoked automatically.

Although double encryption is available in all regions, support was rolled out in two phases. The first rollout was in August 2020 and included the five regions listed below. The second rollout in May 2021 extended double encryption to all remaining regions. If you're using CMK on an older service and want double encryption, you'll need to create a new search service in your region of choice.

Region Service creation date
West US 2 After August 1, 2020
East US After August 1, 2020
South Central US After August 1, 2020
US Gov Virginia After August 1, 2020
US Gov Arizona After August 1, 2020
All other supported regions After May 13, 2021

Prerequisites

The following tools and services are used in this scenario.

You should have a search client that can create the encrypted object. Into this code, you'll reference a key vault key and Active Directory registration information. This code could be a working app, or prototype code such as the C# code sample DotNetHowToEncryptionUsingCMK.

Tip

You can use Postman, Visual Studio Code, or Azure PowerShell, to call REST APIs that create indexes and synonym maps that include an encryption key parameter. You can also use Azure SDKs. Portal support for adding a key to indexes or synonym maps isn't supported.

Key Vault tips

If you're new to Azure Key Vault, review this quickstart to learn about basic tasks: Set and retrieve a secret from Azure Key Vault using PowerShell. Here are some tips for using Key Vault:

  • Use as many key vaults as you need. Managed keys can be in different key vaults. A search service can have multiple encrypted objects, each one encrypted with a different customer-managed encryption key, stored in different key vaults.

  • Enable logging on Key Vault so that you can monitor key usage.

  • Remember to follow strict procedures during routine rotation of key vault keys and Active Directory application secrets and registration. Always update all encrypted content to use new secrets and keys before deleting the old ones. If you miss this step, your content can't be decrypted.

1 - Enable purge protection

As a first step, make sure soft-delete and purge protection are enabled on the key vault. Due to the nature of encryption with customer-managed keys, no one can retrieve your data if your Azure Key Vault key is deleted.

To prevent data loss caused by accidental Key Vault key deletions, soft-delete and purge protection must be enabled on the key vault. Soft-delete is enabled by default, so you'll only encounter issues if you purposely disabled it. Purge protection isn't enabled by default, but it is required for customer-managed key encryption in Cognitive Search.

You can set both properties using the portal, PowerShell, or Azure CLI commands.

  1. Sign in to Azure portal and open your key vault overview page.

  2. On the Overview page under Essentials, enable Soft-delete and Purge protection.

2 - Create a key in Key Vault

Skip key generation if you already have a key in Azure Key Vault that you want to use, but collect the key identifier. You will need this information when creating an encrypted object.

  1. Sign in to Azure portal and open your key vault overview page.

  2. Select Keys on the left, and then select + Generate/Import.

  3. In the Create a key pane, from the list of Options, choose the method that you want to use to create a key. You can Generate a new key, Upload an existing key, or use Restore Backup to select a backup of a key.

  4. Enter a Name for your key, and optionally select other key properties.

  5. Select Create to start the deployment.

  6. Select the key, select the current version, and then make a note of the key identifier. It's composed of the key value Uri, the key name, and the key version. You will need the identifier to define an encrypted index in Azure Cognitive Search.

    Create a new key vault key

3 - Create a security principal

You have several options for accessing the encryption key at run time. The simplest approach is to retrieve the key using the managed identity and permissions of your search service. You can use either a system or user-managed identity. Doing so allows you to omit the steps for application registration and application secrets, and simplifies the encryption key definition.

Alternatively, you can create and register an Azure Active Directory application. The search service will provide the application ID on requests.

A managed identity enables your search service to authenticate to Azure Key Vault without storing credentials (ApplicationID or ApplicationSecret) in code. The lifecycle of this type of managed identity is tied to the lifecycle of your search service, which can only have one managed identity. For more information about how managed identities work, see What are managed identities for Azure resources.

  1. Make your search service a trusted service.

    Turn on system assigned managed identity

Conditions that will prevent you from adopting this approach include:

  • You can't directly grant your search service access permissions to the key vault (for example, if the search service is in a different Active Directory tenant than the Azure Key Vault).

  • A single search service is required to host multiple encrypted indexes or synonym maps, each using a different key from a different key vault, where each key vault must use a different identity for authentication. Because a search service can only have one managed identity, a requirement for multiple identities will disqualify the simplified approach for your scenario.

4 - Grant permissions

In this step, you'll create an access policy in Key Vault. This policy gives the application you registered with Active Directory permission to use your customer-managed key.

Access permissions could be revoked at any given time. Once revoked, any search service index or synonym map that uses that key vault will become unusable. Restoring key vault access permissions at a later time will restore index\synonym map access. For more information, see Secure access to a key vault.

  1. Still in the Azure portal, open your key vault Overview page.

  2. Select the Access policies on the left, and select + Create to start the Create an access policy wizard.

    Create an access policy.

  3. On the Permissions page, select Get for Key permissions, Secret permissions, and Certificate Permissions. Select Unwrap Key and Wrap Key for ** cryptographic operations on the key.

    Select permissions in the Permissions page.

  4. Select Next.

  5. On the Principle page, find and select the security principal used by the search service to access the encryption key. This will either be the system-managed or user-managed identity of the search service, or the registered application.

  6. Select Next and Create.

Important

Encrypted content in Azure Cognitive Search is configured to use a specific Azure Key Vault key with a specific version. If you change the key or version, the index or synonym map must be updated to use it before you delete the previous one. Failing to do so will render the index or synonym map unusable. You won't be able to decrypt the content if the key is lost.

5 - Encrypt content

Encryption keys are added when you create an object. To add a customer-managed key on an index, synonym map, indexer, data source, or skillset, use the Search REST API or an Azure SDK to create an object that has encryption enabled. The portal does not allow encryption properties on object creation.

  1. Call the Create APIs to specify the encryptionKey property:

  2. Insert the encryptionKey construct into the object definition. This property is a first-level property, on the same level as name and description. The REST examples below show property placement. If you're using the same vault, key, and version, you can paste in the same "encryptionKey" construct into each object definition.

    The first example shows an "encryptionKey" for a search service that connects using a managed identity:

    {
      "encryptionKey": {
        "keyVaultUri": "https://demokeyvault.vault.azure.net",
        "keyVaultKeyName": "myEncryptionKey",
        "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660"
      }
    }
    

    The second example includes "accessCredentials", necessary if you registered an application in Azure AD:

    {
      "encryptionKey": {
        "keyVaultUri": "https://demokeyvault.vault.azure.net",
        "keyVaultKeyName": "myEncryptionKey",
        "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
        "accessCredentials": {
          "applicationId": "00000000-0000-0000-0000-000000000000",
          "applicationSecret": "myApplicationSecret"
        }
      }
    }
    

Once you create the encrypted object on the search service, you can use it as you would any other object of its type. Encryption is transparent to the user and developer.

Note

None of these key vault details are considered secret and could be easily retrieved by browsing to the relevant Azure Key Vault page in Azure portal.

REST examples

This section shows the JSON for several objects so that you can see where to locate "encryptionKey" in an object definition.

Index encryption

The details of creating a new index via the REST API could be found at Create Index (REST API), where the only difference here is specifying the encryption key details as part of the index definition:

{
 "name": "hotels",
 "fields": [
  {"name": "HotelId", "type": "Edm.String", "key": true, "filterable": true},
  {"name": "HotelName", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": true, "facetable": false},
  {"name": "Description", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false, "analyzer": "en.lucene"},
  {"name": "Description_fr", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false, "analyzer": "fr.lucene"},
  {"name": "Category", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": true, "facetable": true},
  {"name": "Tags", "type": "Collection(Edm.String)", "searchable": true, "filterable": true, "sortable": false, "facetable": true},
  {"name": "ParkingIncluded", "type": "Edm.Boolean", "filterable": true, "sortable": true, "facetable": true},
  {"name": "LastRenovationDate", "type": "Edm.DateTimeOffset", "filterable": true, "sortable": true, "facetable": true},
  {"name": "Rating", "type": "Edm.Double", "filterable": true, "sortable": true, "facetable": true},
  {"name": "Location", "type": "Edm.GeographyPoint", "filterable": true, "sortable": true}
 ],
  "encryptionKey": {
    "keyVaultUri": "https://demokeyvault.vault.azure.net",
    "keyVaultKeyName": "myEncryptionKey",
    "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
    "accessCredentials": {
      "applicationId": "00000000-0000-0000-0000-000000000000",
      "applicationSecret": "myApplicationSecret"
    }
  }
}

You can now send the index creation request, and then start using the index normally.

Synonym map encryption

Create an encrypted synonym map using the Create Synonym Map Azure Cognitive Search REST API. Use the "encryptionKey" property to specify which encryption key to use.

{
  "name" : "synonymmap1",
  "format" : "solr",
  "synonyms" : "United States, United States of America, USA\n
  Washington, Wash. => WA",
  "encryptionKey": {
    "keyVaultUri": "https://demokeyvault.vault.azure.net",
    "keyVaultKeyName": "myEncryptionKey",
    "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
    "accessCredentials": {
      "applicationId": "00000000-0000-0000-0000-000000000000",
      "applicationSecret": "myApplicationSecret"
    }
  }
}

You can now send the synonym map creation request, and then start using it normally.

Data source encryption

Create an encrypted data source using the Create Data Source (REST API). Use the "encryptionKey" property to specify which encryption key to use.

{
  "name" : "datasource1",
  "type" : "azureblob",
  "credentials" :
  { "connectionString" : "DefaultEndpointsProtocol=https;AccountName=datasource;AccountKey=accountkey;EndpointSuffix=core.windows.net"
  },
  "container" : { "name" : "containername" },
  "encryptionKey": {
    "keyVaultUri": "https://demokeyvault.vault.azure.net",
    "keyVaultKeyName": "myEncryptionKey",
    "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
    "accessCredentials": {
      "applicationId": "00000000-0000-0000-0000-000000000000",
      "applicationSecret": "myApplicationSecret"
    }
  }
}

You can now send the data source creation request, and then start using it normally.

Skillset encryption

Create an encrypted skillset using the Create Skillset REST API. Use the "encryptionKey" property to specify which encryption key to use.

{
    "name": "skillset1",
    "skills":  [ omitted for brevity ],
    "cognitiveServices": { omitted for brevity },
      "knowledgeStore":  { omitted for brevity  },
    "encryptionKey": (optional) { 
        "keyVaultKeyName": "myEncryptionKey",
        "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
        "keyVaultUri": "https://demokeyvault.vault.azure.net",
        "accessCredentials": {
            "applicationId": "00000000-0000-0000-0000-000000000000",
            "applicationSecret": "myApplicationSecret"}
    }
}

You can now send the skillset creation request, and then start using it normally.

Indexer encryption

Create an encrypted indexer using the Create Indexer REST API. Use the "encryptionKey" property to specify which encryption key to use.

{
  "name": "indexer1",
  "dataSourceName": "datasource1",
  "skillsetName": "skillset1",
  "parameters": {
      "configuration": {
          "imageAction": "generateNormalizedImages"
      }
  },
  "encryptionKey": {
    "keyVaultUri": "https://demokeyvault.vault.azure.net",
    "keyVaultKeyName": "myEncryptionKey",
    "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
    "accessCredentials": {
      "applicationId": "00000000-0000-0000-0000-000000000000",
      "applicationSecret": "myApplicationSecret"
    }
  }
}

You can now send the indexer creation request, and then start using it normally.

Important

While "encryptionKey" can't be added to existing search indexes or synonym maps, it may be updated by providing different values for any of the three key vault details (for example, updating the key version). When changing to a new Key Vault key or a new key version, any search index or synonym map that uses the key must first be updated to use the new key\version before deleting the previous key\version. Failing to do so will render the index or synonym map unusable, as it won't be able to decrypt the content once key access is lost. Although restoring key vault access permissions at a later time will restore content access.

Work with encrypted content

With customer-managed key encryption, you'll notice latency for both indexing and queries due to the extra encrypt/decrypt work. Azure Cognitive Search does not log encryption activity, but you can monitor key access through key vault logging. We recommend that you enable logging as part of key vault configuration.

Key rotation is expected to occur over time. Whenever you rotate keys, it's important to follow this sequence:

  1. Determine the key used by an index or synonym map.
  2. Create a new key in key vault, but leave the original key available.
  3. Update the encryptionKey properties on an index or synonym map to use the new values. Only objects that were originally created with this property can be updated to use a different value.
  4. Disable or delete the previous key in the key vault. Monitor key access to verify the new key is being used.

For performance reasons, the search service caches the key for up to several hours. If you disable or delete the key without providing a new one, queries will continue to work on a temporary basis until the cache expires. However, once the search service can no longer decrypt content, you'll get this message: "Access forbidden. The query key used might have been revoked - please retry."

Next steps

If you're unfamiliar with Azure security architecture, review the Azure Security documentation, and in particular, this article: