Key Vault virtual machine extension for Linux

The Key Vault VM extension provides automatic refresh of certificates stored in an Azure key vault. Specifically, the extension monitors a list of observed certificates stored in key vaults. Upon detecting a change, the extension retrieves, and installs the corresponding certificates. The Key Vault VM extension is published and supported by Microsoft, currently on Linux VMs. This document details the supported platforms, configurations, and deployment options for the Key Vault VM extension for Linux.

Operating system

The Key Vault VM extension supports these Linux distributions:

  • Ubuntu-1804
  • Suse-15

Note

To get extended security features, prepare to upgrade Ubuntu-1604 and Debian-9 systems as these versions are reaching their end of designated support period.

Supported certificate content types

  • PKCS #12
  • PEM

Prerequisities

  • Key Vault instance with certificate. See Create a Key Vault

  • VM/VMSS must have assigned managed identity

  • The Key Vault Access Policy must be set with secrets get and list permission for VM/VMSS managed identity to retrieve a secret's portion of certificate. See How to Authenticate to Key Vault and Assign a Key Vault access policy.

  • VMSS should have the following identity setting: "identity": { "type": "UserAssigned", "userAssignedIdentities": { "[parameters('userAssignedIdentityResourceId')]": {} } }

  • AKV extension should have this setting: "authenticationSettings": { "msiEndpoint": "[parameters('userAssignedIdentityEndpoint')]", "msiClientId": "[reference(parameters('userAssignedIdentityResourceId'), variables('msiApiVersion')).clientId]" }

Key Vault VM extension version

  • Ubuntu-18.04 and SUSE-15 users can chose to upgrade their key vault vm extension version to V2.0 to avail full certificate chain download feature. Issuer certificates (intermediate and root) will be appended to the leaf certificate in the PEM file.

  • If you prefer to upgrade to v2.0, you would need to delete v1.0 first, then install v2.0.

  az vm extension delete --name KeyVaultForLinux --resource-group ${resourceGroup} --vm-name ${vmName}
  az vm extension set -n "KeyVaultForLinux" --publisher Microsoft.Azure.KeyVault --resource-group "${resourceGroup}" --vm-name "${vmName}" –settings .\akvvm.json –version 2.0

The flag --version 2.0 is optional because the latest version will be installed by default.

  • If the VM has certificates downloaded by v1.0, deleting the v1.0 AKVVM extension will NOT delete the downloaded certificates. After installing v2.0, the existing certificates will NOT be modified. You would need to delete the certificate files or roll-over the certificate to get the PEM file with full-chain on the VM.

Extension schema

The following JSON shows the schema for the Key Vault VM extension. The extension does not require protected settings - all its settings are considered information without security impact. The extension requires a list of monitored secrets, polling frequency, and the destination certificate store. Specifically:

    {
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "KVVMExtensionForLinux",
      "apiVersion": "2019-07-01",
      "location": "<location>",
      "dependsOn": [
          "[concat('Microsoft.Compute/virtualMachines/', <vmName>)]"
      ],
      "properties": {
      "publisher": "Microsoft.Azure.KeyVault",
      "type": "KeyVaultForLinux",
      "typeHandlerVersion": "2.0",
      "autoUpgradeMinorVersion": true,
      "settings": {
        "secretsManagementSettings": {
          "pollingIntervalInS": <polling interval in seconds, e.g. "3600">,
          "certificateStoreName": <It is ignored on Linux>,
          "linkOnRenewal": <Not available on Linux e.g.: false>,
          "certificateStoreLocation": <disk path where certificate is stored, default: "/var/lib/waagent/Microsoft.Azure.KeyVault">,
          "requireInitialSync": <initial synchronization of certificates e..g: true>,
          "observedCertificates": <list of KeyVault URIs representing monitored certificates, e.g.: ["https://myvault.vault.azure.net/secrets/mycertificate", "https://myvault.vault.azure.net/secrets/mycertificate2"]>
        },
        "authenticationSettings": {
                "msiEndpoint":  <Optional MSI endpoint e.g.: "http://169.254.169.254/metadata/identity">,
                "msiClientId":  <Optional MSI identity e.g.: "c7373ae5-91c2-4165-8ab6-7381d6e75619">
        }
       }
      }
    }

Note

Your observed certificates URLs should be of the form https://myVaultName.vault.azure.net/secrets/myCertName.

This is because the /secrets path returns the full certificate, including the private key, while the /certificates path does not. More information about certificates can be found here: Key Vault Certificates

Important

The 'authenticationSettings' property is required only for VMs with user assigned identities. It specifies identity to use for authentication to Key Vault.

Property values

Name Value / Example Data Type
apiVersion 2019-07-01 date
publisher Microsoft.Azure.KeyVault string
type KeyVaultForLinux string
typeHandlerVersion 2.0 int
pollingIntervalInS 3600 string
certificateStoreName It is ignored on Linux string
linkOnRenewal false boolean
certificateStoreLocation /var/lib/waagent/Microsoft.Azure.KeyVault string
requireInitialSync true boolean
observedCertificates ["https://myvault.vault.azure.net/secrets/mycertificate", "https://myvault.vault.azure.net/secrets/mycertificate2"] string array
msiEndpoint http://169.254.169.254/metadata/identity string
msiClientId c7373ae5-91c2-4165-8ab6-7381d6e75619 string

Template deployment

Azure VM extensions can be deployed with Azure Resource Manager templates. Templates are ideal when deploying one or more virtual machines that require post deployment refresh of certificates. The extension can be deployed to individual VMs or virtual machine scale sets. The schema and configuration are common to both template types.

The JSON configuration for a virtual machine extension must be nested inside the virtual machine resource fragment of the template, specifically "resources": [] object for the virtual machine template and in case of virtual machine scale set under "virtualMachineProfile":"extensionProfile":{"extensions" :[] object.

Note

The VM extension would require system or user managed identity to be assigned to authenticate to Key vault. See How to authenticate to Key Vault and assign a Key Vault access policy.

    {
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "KeyVaultForLinux",
      "apiVersion": "2019-07-01",
      "location": "<location>",
      "dependsOn": [
          "[concat('Microsoft.Compute/virtualMachines/', <vmName>)]"
      ],
      "properties": {
      "publisher": "Microsoft.Azure.KeyVault",
      "type": "KeyVaultForLinux",
      "typeHandlerVersion": "2.0",
      "autoUpgradeMinorVersion": true,
      "settings": {
          "secretsManagementSettings": {
          "pollingIntervalInS": <polling interval in seconds, e.g. "3600">,
          "certificateStoreName": <ingnored on linux>,
          "certificateStoreLocation": <disk path where certificate is stored, default: "/var/lib/waagent/Microsoft.Azure.KeyVault">,
          "observedCertificates": <list of KeyVault URIs representing monitored certificates, e.g.: "https://myvault.vault.azure.net/secrets/mycertificate"
        }      
      }
      }
    }

Extension Dependency Ordering

The Key Vault VM extension supports extension ordering if configured. By default the extension reports that it has successfully started as soon as it has started polling. However, it can be configured to wait until it has successfully downloaded the complete list of certificates before reporting a successful start. If other extensions depend on having the full set of certificates install before they start, then enabling this setting will allow those extension to declare a dependency on the Key Vault extension. This will prevent those extensions from starting until all certificates they depend on have been installed. The extension will retry the initial download indefinitely and remain in a Transitioning state.

To turn this on set the following:

"secretsManagementSettings": {
    "requireInitialSync": true,
    ...
}

[Note] Using this feature is not compatible with an ARM template that creates a system assigned identity and updates a Key Vault access policy with that identity. Doing so will result in a deadlock as the vault access policy cannot be updated until all extensions have started. You should instead use a single user assigned MSI identity and pre-ACL your vaults with that identity before deploying.

Azure PowerShell deployment

Warning

PowerShell clients often add \ to " in the settings.json which will cause akvvm_service fails with error: [CertificateManagementConfiguration] Failed to parse the configuration settings with:not an object.

The Azure PowerShell can be used to deploy the Key Vault VM extension to an existing virtual machine or virtual machine scale set.

  • To deploy the extension on a VM:

        # Build settings
        $settings = '{"secretsManagementSettings": 
        { "pollingIntervalInS": "' + <pollingInterval> + 
        '", "certificateStoreName": "' + <certStoreName> + 
        '", "certificateStoreLocation": "' + <certStoreLoc> + 
        '", "observedCertificates": ["' + <observedCert1> + '","' + <observedCert2> + '"] } }'
        $extName =  "KeyVaultForLinux"
        $extPublisher = "Microsoft.Azure.KeyVault"
        $extType = "KeyVaultForLinux"
    
    
        # Start the deployment
        Set-AzVmExtension -TypeHandlerVersion "2.0" -ResourceGroupName <ResourceGroupName> -Location <Location> -VMName <VMName> -Name $extName -Publisher $extPublisher -Type $extType -SettingString $settings
    
    
  • To deploy the extension on a virtual machine scale set :

    
        # Build settings
        $settings = '{"secretsManagementSettings": 
        { "pollingIntervalInS": "' + <pollingInterval> + 
        '", "certificateStoreName": "' + <certStoreName> + 
        '", "certificateStoreLocation": "' + <certStoreLoc> + 
        '", "observedCertificates": ["' + <observedCert1> + '","' + <observedCert2> + '"] } }'
        $extName = "KeyVaultForLinux"
        $extPublisher = "Microsoft.Azure.KeyVault"
        $extType = "KeyVaultForLinux"
    
        # Add Extension to VMSS
        $vmss = Get-AzVmss -ResourceGroupName <ResourceGroupName> -VMScaleSetName <VmssName>
        Add-AzVmssExtension -VirtualMachineScaleSet $vmss  -Name $extName -Publisher $extPublisher -Type $extType -TypeHandlerVersion "2.0" -Setting $settings
    
        # Start the deployment
        Update-AzVmss -ResourceGroupName <ResourceGroupName> -VMScaleSetName <VmssName> -VirtualMachineScaleSet $vmss 
    
    

Azure CLI deployment

The Azure CLI can be used to deploy the Key Vault VM extension to an existing virtual machine or virtual machine scale set.

  • To deploy the extension on a VM:

       # Start the deployment
         az vm extension set -n "KeyVaultForLinux" `
         --publisher Microsoft.Azure.KeyVault `
         -g "<resourcegroup>" `
         --vm-name "<vmName>" `
         --version 2.0 `
         --settings '{\"secretsManagementSettings\": { \"pollingIntervalInS\": \"<pollingInterval>\", \"certificateStoreName\": \"<certStoreName>\", \"certificateStoreLocation\": \"<certStoreLoc>\", \"observedCertificates\": [\" <observedCert1> \", \" <observedCert2> \"] }}'
    
  • To deploy the extension on a virtual machine scale set :

          # Start the deployment
          az vmss extension set -n "KeyVaultForLinux" `
          --publisher Microsoft.Azure.KeyVault `
          -g "<resourcegroup>" `
          --vmss-name "<vmssName>" `
          --version 2.0 `
          --settings '{\"secretsManagementSettings\": { \"pollingIntervalInS\": \"<pollingInterval>\", \"certificateStoreName\": \"<certStoreName>\", \"certificateStoreLocation\": \"<certStoreLoc>\", \"observedCertificates\": [\" <observedCert1> \", \" <observedCert2> \"] }}'
    

Please be aware of the following restrictions/requirements:

Frequently Asked Questions

  • Is there is a limit on the number of observedCertificates you can setup? No, Key Vault VM Extension doesn’t have limit on the number of observedCertificates.

Troubleshoot

Data about the state of extension deployments can be retrieved from the Azure portal, and by using the Azure PowerShell. To see the deployment state of extensions for a given VM, run the following command using the Azure PowerShell.

Azure PowerShell

Get-AzVMExtension -VMName <vmName> -ResourceGroupname <resource group name>

Azure CLI

 az vm get-instance-view --resource-group <resource group name> --name  <vmName> --query "instanceView.extensions"

Logs and configuration

/var/log/waagent.log
/var/log/azure/Microsoft.Azure.KeyVault.KeyVaultForLinux/*
/var/lib/waagent/Microsoft.Azure.KeyVault.KeyVaultForLinux-<most recent version>/config/*

Symbolic links or Symlinks are basically advanced shortcuts. To avoid monitoring the folder and to get the latest certificate automatically, you can use this symlink ([VaultName].[CertificateName]) to get the latest version of certificate on Linux.

Frequently Asked Questions

  • Is there is a limit on the number of observedCertificates you can setup? No, Key Vault VM Extension doesn’t have limit on the number of observedCertificates.

Support

If you need more help at any point in this article, you can contact the Azure experts on the MSDN Azure and Stack Overflow forums. Alternatively, you can file an Azure support incident. Go to the Azure support site and select Get support. For information about using Azure Support, read the Microsoft Azure support FAQ.