Plug in CA certificates for Istio-based service mesh add-on on Azure Kubernetes Service
In the Istio-based service mesh addon for Azure Kubernetes Service, by default the Istio certificate authority (CA) generates a self-signed root certificate and key and uses them to sign the workload certificates. To protect the root CA key, you should use a root CA, which runs on a secure machine offline. You can use the root CA to issue intermediate certificates to the Istio CAs that run in each cluster. An Istio CA can sign workload certificates using the administrator-specified certificate and key, and distribute an administrator-specified root certificate to the workloads as the root of trust. This article addresses how to bring your own certificates and keys for Istio CA in the Istio-based service mesh add-on for Azure Kubernetes Service.
This article addresses how you can configure the Istio certificate authority with a root certificate, signing certificate and key provided as inputs using Azure Key Vault to the Istio-based service mesh add-on.
Before you begin
Verify Azure CLI version
The add-on requires Azure CLI version 2.57.0 or later installed. You can run az --version to verify version. To install or upgrade, see [Install Azure CLI][azure-cli-install].
Set up Azure Key Vault
You need an Azure Key Vault resource to supply the certificate and key inputs to the Istio add-on.
You need to generate root certificate, intermediate certificates, intermediate key, and the certificate chain offline. Steps 1-3 from here has an example of how to generate these files.
Create secrets in Azure Key Vault using the certificates and key:
az keyvault secret set --vault-name $AKV_NAME --name root-cert --file <path-to-folder/root-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-cert --file <path-to-folder/ca-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-key --file <path-to-folder/ca-key.pem> az keyvault secret set --vault-name $AKV_NAME --name cert-chain --file <path/cert-chain.pem>Enable Azure Key Vault provider for Secret Store CSI Driver for your cluster:
az aks enable-addons --addons azure-keyvault-secrets-provider --resource-group $RESOURCE_GROUP --name $CLUSTERNote
When rotating certificates, to control how quickly the secrets are synced down to the cluster you can use the
--rotation-poll-intervalparameter of the Azure Key Vault Secrets Provider add-on. For example:az aks addon update --resource-group $RESOURCE_GROUP --name $CLUSTER --addon azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 20sAuthorize the user-assigned managed identity of the add-on to have access to the Azure Key Vault resource:
OBJECT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.objectId' -o tsv) az keyvault set-policy --name $AKV_NAME --object-id $OBJECT_ID --secret-permissions get listNote
If you created your Key Vault with Azure RBAC Authorization for your permission model instead of Vault Access Policy, follow the instructions here to create permissions for the managed identity. Add an Azure role assignment for
Key Vault Readerfor the add-on's user-assigned managed identity.
Set up Istio-based service mesh addon with plug-in CA certificates
Enable the Istio service mesh addon for your existing AKS cluster while referencing the Azure Key Vault secrets that were created earlier:
az aks mesh enable --resource-group $RESOURCE_GROUP --name $CLUSTER \ --root-cert-object-name root-cert \ --ca-cert-object-name ca-cert \ --ca-key-object-name ca-key \ --cert-chain-object-name cert-chain \ --key-vault-id /subscriptions/$SUBSCRIPTION/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$AKV_NAMENote
For existing clusters with Istio addon using self-signed root certificate generated by Istio CA, switching to plugin CA is not supported. You need to disable the mesh on these clusters first and then enable it again using the above command to pass through the plugin CA inputs.
Verify that the
cacertsgets created on the cluster:kubectl get secret -n aks-istio-systemExpected output:
NAME TYPE DATA AGE cacerts opaque 4 13h sh.helm.release.v1.azure-service-mesh-istio-discovery.v380 helm.sh/release.v1 1 2m15s sh.helm.release.v1.azure-service-mesh-istio-discovery.v381 helm.sh/release.v1 1 8sVerify that the Istio control plane picked up the custom certificate authority:
kubectl logs deploy/istiod-asm-1-17 -c discovery -n aks-istio-system | grep -v validationController | grep x509Expected output should be similar to:
2023-11-06T15:49:15.493732Z info x509 cert - Issuer: "CN=Intermediate CA - A1,O=Istio,L=cluster-A1", Subject: "", SN: e191d220af347c7e164ec418d75ed19e, NotBefore: "2023-11-06T15:47:15Z", NotAfter: "2033-11-03T15:49:15Z" 2023-11-06T15:49:15.493764Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Intermediate CA - A1,O=Istio,L=cluster-A1", SN: 885034cba2894f61036f2956fd9d0ed337dc636, NotBefore: "2023-11-04T01:40:02Z", NotAfter: "2033-11-01T01:40:02Z" 2023-11-06T15:49:15.493795Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Root A,O=Istio", SN: 18e2ee4089c5a7363ec306627d21d9bb212bed3e, NotBefore: "2023-11-04T01:38:27Z", NotAfter: "2033-11-01T01:38:27Z"
Certificate authority rotation
You may need to periodically rotate the certificate authorities for security or policy reasons. This section walks you through how to handle intermediate CA and root CA rotation scenarios.
Intermediate certificate authority rotation
You can rotate the intermediate CA while keeping the root CA the same. Update the secrets in Azure Key Vault resource with the new certificate and key files:
az keyvault secret set --vault-name $AKV_NAME --name root-cert --file <path-to-folder/root-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-cert --file <path-to-folder/ca-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-key --file <path-to-folder/ca-key.pem> az keyvault secret set --vault-name $AKV_NAME --name cert-chain --file <path/cert-chain.pem>Wait for the time duration of
--rotation-poll-interval. Check if thecacertssecret was refreshed on the cluster based on the new intermediate CA that was updated on the Azure Key Vault resource:kubectl logs deploy/istiod-asm-1-17 -c discovery -n aks-istio-system | grep -v validationControllerExpected output should be similar to:
2023-11-07T06:16:21.091844Z info Update Istiod cacerts 2023-11-07T06:16:21.091901Z info Using istiod file format for signing ca files 2023-11-07T06:16:21.354423Z info Istiod has detected the newly added intermediate CA and updated its key and certs accordingly 2023-11-07T06:16:21.354910Z info x509 cert - Issuer: "CN=Intermediate CA - A2,O=Istio,L=cluster-A2", Subject: "", SN: b2753c6a23b54d8364e780bf664672ce, NotBefore: "2023-11-07T06:14:21Z", NotAfter: "2033-11-04T06:16:21Z" 2023-11-07T06:16:21.354967Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Intermediate CA - A2,O=Istio,L=cluster-A2", SN: 17f36ace6496ac2df88e15878610a0725bcf8ae9, NotBefore: "2023-11-04T01:40:22Z", NotAfter: "2033-11-01T01:40:22Z" 2023-11-07T06:16:21.355007Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Root A,O=Istio", SN: 18e2ee4089c5a7363ec306627d21d9bb212bed3e, NotBefore: "2023-11-04T01:38:27Z", NotAfter: "2033-11-01T01:38:27Z" 2023-11-07T06:16:21.355012Z info Istiod certificates are reloadedThe workloads receive certificates from the Istio control plane that are valid for 24 hours by default. If you don't restart the pods, all the workloads obtain new leaf certificates based on the new intermediate CA in 24 hours. If you want to force all these workloads to obtain new leaf certificates right away from the new intermediate CA, then you need to restart the workloads.
kubectl rollout restart deployment <deployment name> -n <deployment namespace>
Root certificate authority rotation
You need to update Azure Key Vault secrets with the root certificate file having the concatenation of the old and the new root certificates:
az keyvault secret set --vault-name $AKV_NAME --name root-cert --file <path-to-folder/root-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-cert --file <path-to-folder/ca-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-key --file <path-to-folder/ca-key.pem> az keyvault secret set --vault-name $AKV_NAME --name cert-chain --file <path/cert-chain.pem>Contents of
root-cert.pemfollow this format:-----BEGIN CERTIFICATE----- <contents of old root certificate> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <contents of new root certificate> -----END CERTIFICATE-----The add-on includes a
CronJobrunning every ten minutes on the cluster to check for updates to root certificate. If it detects an update, it restarts the Istio control plane (istioddeployment) to pick up the updates. You can check its logs to confirm that the root certificate update was detected and that the Istio control plane was restarted:kubectl logs -n aks-istio-system $(kubectl get pods -n aks-istio-system | grep 'istio-cert-validator-cronjob-' | sort -k8 | tail -n 1 | awk '{print $1}')Expected output:
Root certificate update detected. Restarting deployment... deployment.apps/istiod-asm-1-17 restarted Deployment istiod-asm-1-17 restarted.After
istiodwas restarted, it should indicate that two certificates were added to the trust domain:kubectl logs deploy/istiod-asm-1-17 -c discovery -n aks-istio-systemExpected output:
2023-11-07T06:42:00.287916Z info Using istiod file format for signing ca files 2023-11-07T06:42:00.287928Z info Use plugged-in cert at etc/cacerts/ca-key.pem 2023-11-07T06:42:00.288254Z info x509 cert - Issuer: "CN=Intermediate CA - A2,O=Istio,L=cluster-A2", Subject: "", SN: 286451ca8ff7bf9e6696f56bef829d42, NotBefore: "2023-11-07T06:40:00Z", NotAfter: "2033-11-04T06:42:00Z" 2023-11-07T06:42:00.288279Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Intermediate CA - A2,O=Istio,L=cluster-A2", SN: 17f36ace6496ac2df88e15878610a0725bcf8ae9, NotBefore: "2023-11-04T01:40:22Z", NotAfter: "2033-11-01T01:40:22Z" 2023-11-07T06:42:00.288298Z info x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Root A,O=Istio", SN: 18e2ee4089c5a7363ec306627d21d9bb212bed3e, NotBefore: "2023-11-04T01:38:27Z", NotAfter: "2033-11-01T01:38:27Z" 2023-11-07T06:42:00.288303Z info Istiod certificates are reloaded 2023-11-07T06:42:00.288365Z info spiffe Added 2 certs to trust domain cluster.local in peer cert verifierYou need to either wait for 24 hours (the default time for leaf certificate validity) or force a restart of all the workloads. This way, all workloads recognize both the old and the new certificate authorities for mTLS verification.
kubectl rollout restart deployment <deployment name> -n <deployment namespace>You can now update Azure Key Vault secrets with only the new CA (without the old CA):
az keyvault secret set --vault-name $AKV_NAME --name root-cert --file <path-to-folder/root-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-cert --file <path-to-folder/ca-cert.pem> az keyvault secret set --vault-name $AKV_NAME --name ca-key --file <path-to-folder/ca-key.pem> az keyvault secret set --vault-name $AKV_NAME --name cert-chain --file <path/cert-chain.pem>Check the logs of the
CronJobto confirm detection of root certificate update and the restart ofistiod:kubectl logs -n aks-istio-system $(kubectl get pods -n aks-istio-system | grep 'istio-cert-validator-cronjob-' | sort -k8 | tail -n 1 | awk '{print $1}')Expected output:
Root certificate update detected. Restarting deployment... deployment.apps/istiod-asm-1-17 restarted Deployment istiod-asm-1-17 restarted.After
istiodwas updated, it should only confirm the usage of new root CA:kubectl logs deploy/istiod-asm-1-17 -c discovery -n aks-istio-system | grep -v validationControllerExpected output:
2023-11-07T08:01:17.780299Z info x509 cert - Issuer: "CN=Intermediate CA - B1,O=Istio,L=cluster-B1", Subject: "", SN: 1159747c72cc7ac7a54880cd49b8df0a, NotBefore: "2023-11-07T07:59:17Z", NotAfter: "2033-11-04T08:01:17Z" 2023-11-07T08:01:17.780330Z info x509 cert - Issuer: "CN=Root B,O=Istio", Subject: "CN=Intermediate CA - B1,O=Istio,L=cluster-B1", SN: 2aba0c438652a1f9beae4249457023013948c7e2, NotBefore: "2023-11-04T01:42:12Z", NotAfter: "2033-11-01T01:42:12Z" 2023-11-07T08:01:17.780345Z info x509 cert - Issuer: "CN=Root B,O=Istio", Subject: "CN=Root B,O=Istio", SN: 3f9da6ddc4cb03749c3f43243a4b701ce5eb4e96, NotBefore: "2023-11-04T01:41:54Z", NotAfter: "2033-11-01T01:41:54Z"From the example outputs shown in this article, you can observe that we moved from Root A (used when enabling the addon) to Root B.
You can either again wait for 24 hours or force a restart of all the workloads. Forcing a restart makes the workloads obtain new leaf certificates from the new root CA immediately.
kubectl rollout restart deployment <deployment name> -n <deployment namespace>
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for