Povratne informacije Uredi

Using Azure Key Vault Secrets Provider extension to fetch secrets into Arc clusters (Preview)

  • Članak
  • 7 min. za čitanje

The Azure Key Vault Provider for Secrets Store CSI Driver allows for the integration of Azure Key Vault as a secrets store with a Kubernetes cluster via a CSI volume.

Prerequisites

  1. Ensure you have met all the common prerequisites for cluster extensions listed here.
  2. Use az k8s-extension CLI version >= v0.4.0

Support limitations for Azure Key Vault (AKV) secrets provider extension

  • Following Kubernetes distributions are currently supported
    • Cluster API Azure
    • Azure Kubernetes Service on Azure Stack HCI (AKS-HCI)
    • Google Kubernetes Engine
    • OpenShift Kubernetes Distribution
    • Canonical Kubernetes Distribution
    • Elastic Kubernetes Service

Important

Azure Arc-enabled Kubernetes preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Arc-enabled Kubernetes previews are partially covered by customer support on a best-effort basis.

Features

  • Mounts secrets/keys/certs to pod using a CSI Inline volume
  • Supports pod portability with the SecretProviderClass CRD
  • Supports Linux and Windows containers
  • Supports sync with Kubernetes Secrets
  • Supports auto rotation of secrets

Install AKV secrets provider extension on an Arc enabled Kubernetes cluster

The following steps assume that you already have a cluster with supported Kubernetes distribution connected to Azure Arc.

Set the environment variables:

export CLUSTER_NAME=<arc-cluster-name>
export RESOURCE_GROUP=<resource-group-name>

While AKV secrets provider extension is in preview, the az k8s-extension create command only accepts preview for the --release-train flag.

az k8s-extension create --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --cluster-type connectedClusters --extension-type Microsoft.AzureKeyVaultSecretsProvider --release-train preview --name akvsecretsprovider

The above will install the Secrets Store CSI Driver and the Azure Key Vault Provider on your cluster nodes. You should see output similar to the output shown below. It may take 3-5 minutes for the actual AKV secrets provider helm chart to get deployed to the cluster.

Note that only one instance of AKV secrets provider extension can be deployed on an Arc connected Kubernetes cluster.

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Kubernetes/connectedClusters/$CLUSTER_NAME/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "location": null,
  "name": "sscsi",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "preview",
  "resourceGroup": "$RESOURCE_GROUP",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2021-11-15T18:55:33.952130+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2021-11-15T18:55:33.952130+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.0.0"
}

Install AKV secrets provider extension using ARM template

After connecting your cluster to Azure Arc, create a json file with the following format, making sure to update the <cluster-name> value:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "ConnectedClusterName": {
            "defaultValue": "<cluster-name>",
            "type": "String",
            "metadata": {
                "description": "The Connected Cluster name."
            }
        },
        "ExtensionInstanceName": {
            "defaultValue": "akvsecretsprovider",
            "type": "String",
            "metadata": {
                "description": "The extension instance name."
            }
        },
        "ExtensionVersion": {
            "defaultValue": "",
            "type": "String",
            "metadata": {
                "description": "The version of the extension type."
            }
        },
        "ExtensionType": {
            "defaultValue": "Microsoft.AzureKeyVaultSecretsProvider",
            "type": "String",
            "metadata": {
                "description": "The extension type."
            }
        },
        "ReleaseTrain": {
            "defaultValue": "preview",
            "type": "String",
            "metadata": {
                "description": "The release train."
            }
        }
    },
    "functions": [],
    "resources": [
        {
            "type": "Microsoft.KubernetesConfiguration/extensions",
            "apiVersion": "2021-09-01",
            "name": "[parameters('ExtensionInstanceName')]",
            "properties": {
                "extensionType": "[parameters('ExtensionType')]",
                "releaseTrain": "[parameters('ReleaseTrain')]",
                "version": "[parameters('ExtensionVersion')]"
            },
            "scope": "[concat('Microsoft.Kubernetes/connectedClusters/', parameters('ConnectedClusterName'))]"
        }
    ]
}

Now set the environment variables:

export TEMPLATE_FILE_NAME=<template-file-path>
export DEPLOYMENT_NAME=<desired-deployment-name>

Finally, run this command to install the AKV secrets provider extension through az CLI:

az deployment group create --name $DEPLOYMENT_NAME --resource-group $RESOURCE_GROUP --template-file $TEMPLATE_FILE_NAME

Now, you should be able to view the AKV provider resources and use the extension in your cluster.

Validate the extension installation

Run the following command.

az k8s-extension show --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name akvsecretsprovider

You should see a JSON output similar to the output below:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Kubernetes/connectedClusters/$CLUSTER_NAME/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "location": null,
  "name": "akvsecretsprovider",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "preview",
  "resourceGroup": "$RESOURCE_GROUP",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2021-11-15T21:17:52.751916+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2021-11-15T21:17:52.751916+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.0.0"
}

Create or use an existing Azure Key Vault

Set the environment variables:

export AKV_RESOURCE_GROUP=<resource-group-name>
export AZUREKEYVAULT_NAME=<AKV-name>
export AZUREKEYVAULT_LOCATION=<AKV-location>

You will need an Azure Key Vault resource containing the secret content. Keep in mind that the Key Vault's name must be globally unique.

az keyvault create -n $AZUREKEYVAULT_NAME -g $AKV_RESOURCE_GROUP -l $AZUREKEYVAULT_LOCATION

Azure Key Vault can store keys, secrets, and certificates. In this example, we'll set a plain text secret called DemoSecret:

az keyvault secret set --vault-name $AZUREKEYVAULT_NAME -n DemoSecret --value MyExampleSecret

Take note of the following properties for use in the next section:

  • Name of secret object in Key Vault
  • Object type (secret, key, or certificate)
  • Name of your Azure Key Vault resource
  • Azure Tenant ID the Subscription belongs to

Provide identity to access Azure Key Vault

The Secrets Store CSI Driver on Arc connected clusters currently allows for the following methods to access an Azure Key Vault instance:

  • Service Principal

Follow the steps below to provide identity to access Azure Key Vault

  1. Follow the steps here to create a service principal in Azure. Take note of the Client ID and Client Secret generated in this step.
  2. Provide Azure Key Vault GET permission to the created service principal by following the steps here.
  3. Use the client ID and Client Secret from step 1 to create a Kubernetes secret on the Arc connected cluster:
kubectl create secret generic secrets-store-creds --from-literal clientid="<client-id>" --from-literal clientsecret="<client-secret>"
  1. Label the created secret:
kubectl label secret secrets-store-creds secrets-store.csi.k8s.io/used=true
  1. Create a SecretProviderClass with the following YAML, filling in your values for key vault name, tenant ID, and objects to retrieve from your AKV instance:
# This is a SecretProviderClass example using service principal to access Keyvault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: akvprovider-demo
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    keyvaultName: <key-vault-name>
    objects:  |
      array:
        - |
          objectName: DemoSecret
          objectType: secret             # object types: secret, key or cert
          objectVersion: ""              # [OPTIONAL] object versions, default to latest if empty
    tenantId: <tenant-Id>                # The tenant ID of the Azure Key Vault instance
  1. Apply the SecretProviderClass to your cluster:
kubectl apply -f secretproviderclass.yaml
  1. Create a pod with the following YAML, filling in the name of your identity:
# This is a sample pod definition for using SecretProviderClass and service principal to access Keyvault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "akvprovider-demo"
        nodePublishSecretRef:                       
          name: secrets-store-creds
  1. Apply the pod to your cluster:
kubectl apply -f pod.yaml

Validate the secrets

After the pod starts, the mounted content at the volume path specified in your deployment YAML is available.

## show secrets held in secrets-store
kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/

## print a test secret 'DemoSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/DemoSecret

Additional configuration options

Following configuration settings are available for Azure Key Vault secrets provider extension:

Configuration Setting Default Description
enableSecretRotation false Boolean type; Periodically update the pod mount and Kubernetes Secret with the latest content from external secrets store
rotationPollInterval 2m Secret rotation poll interval duration if enableSecretRotation is true. This can be tuned based on how frequently the mounted contents for all pods and Kubernetes secrets need to be resynced to the latest
syncSecret.enabled false Boolean input; In some cases, you may want to create a Kubernetes Secret to mirror the mounted content. This configuration setting allows SecretProviderClass to allow secretObjects field to define the desired state of the synced Kubernetes secret objects

These settings can be changed either at the time of extension installation using az k8s-extension create command or post installation using az k8s-extension update command.

Use following command to add configuration settings while creating extension instance:

az k8s-extension create --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --cluster-type connectedClusters --extension-type Microsoft.AzureKeyVaultSecretsProvider --release-train preview --name akvsecretsprovider --configuration-settings secrets-store-csi-driver.enableSecretRotation=true secrets-store-csi-driver.rotationPollInterval=3m secrets-store-csi-driver.syncSecret.enabled=true

Use following command to update configuration settings of existing extension instance:

az k8s-extension update --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --cluster-type connectedClusters --name akvsecretsprovider --configuration-settings secrets-store-csi-driver.enableSecretRotation=true secrets-store-csi-driver.rotationPollInterval=3m secrets-store-csi-driver.syncSecret.enabled=true

Uninstall Azure Key Vault secrets provider extension

Use the below command:

az k8s-extension delete --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name akvsecretsprovider

Note that the uninstallation does not delete the CRDs that are created at the time of extension installation.

Verify that the extension instance has been deleted.

az k8s-extension list --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP

This output should not include AKV secrets provider. If you don't have any other extensions installed on your cluster, it will just be an empty array.

Reconciliation and Troubleshooting

Azure Key Vault secrets provider extension is self-healing. All extension components that are deployed on the cluster at the time of extension installation are reconciled to their original state in case somebody tries to intentionally or unintentionally change or delete them. The only exception to that is CRDs. In case the CRDs are deleted, they are not reconciled. You can bring them back by using the 'az k8s-exstension create' command again and providing the existing extension instance name.

Some common issues and troubleshooting steps for Azure Key Vault secrets provider are captured in the open source documentation here for your reference.

Additional troubleshooting steps that are specific to the Secrets Store CSI Driver Interface can be referenced here.

Frequently asked questions

Is the extension of Azure Key Vault Secrets Provider zone redundant?

Yes, all components of Azure Key Vault Secrets Provider are deployed on availability zones and are hence zone redundant.

Next steps

Just want to try things out?
Get started quickly with an Azure Arc Jumpstart scenario using Cluster API.