Rotate Kubernetes certificates on Azure Stack Hub
This document provides guidance on how to rotate certificates on an existing AKS Engine cluster and recommendations for using adopting
aks-engine rotate-certs as a tool.
This guide assumes that you already have deployed a cluster using AKS engine and the cluster is in a healthy state.
Planning for certificate rotation
When considering using this functionality, be aware that the Kubernetes control plane will be unavailable during the update, validation, and restart steps. Plan this maintenance operation accordingly. Also, plan to execute this operation in a staging environment with equal configuration to the production environment before trying in production.
Review the following considerations before attempting this operation:
You will need access to the API model (
apimodel.json) that was generated by the commands
aks-engine generate. By default this file is placed into a relative directory such as
aks-engine rotate-certsoperation causes API server downtime.
aks-engine rotate-certsexpects an API model that conforms to the current state of the cluster.
aks-engine rotate-certsexecutes remote commands on the cluster nodes and uses the API model information to establish a secure SSH connection.
aks-engine rotate-certsalso relies on some resources to be named in accordance with the original
aks-enginedeployment, for example, VMs must follow the naming provided by
aks-engine rotate-certsrelies upon a working connection to the cluster control plane during certificate rotation:
- To validate each step of the process.
- To restart/recreate cluster resources such as kube-system pods and service account tokens.
If you are rotating the certificates of a cluster in a VNet closed to outside access, you must run
aks-engine rotate-certsfrom a host VM that has network access to the control plane, for example, a jumpbox VM that resides in the same VNet as the master VMs.
If you are using
aks-engine rotate-certsin production, it is recommended to stage a certificate rotation test on a cluster that was built to the same specifications. That is, the cluster is built with the same cluster configuration, the same version of the AKS engine command-line tool, and the same set of enabled addons as your production cluster before performing the certificate rotation. The AKS engine supports different cluster configurations and the extent of end-to-end testing that the AKS engine team runs cannot practically cover every possible configuration. Therefore, it is recommended that you ensure in a staging environment that your specific cluster configuration works with
aks-engine rotate-certsbefore attempting the operation on your production cluster.
aks-engine rotate-certsdoes not guarantee backwards compatibility. If you deployed with aks-engine version 0.60.x, you should prefer executing the certificate rotation process with version 0.60.x.
Fetching a new set of certificates from Key Vault is not supported at this point.
Use a reliable network connection.
aks-engine rotate-certsrequires the execution of multiple remote commands, which are subject to potential failures, mostly if the connection to the cluster nodes is not reliable. Running
aks-engine rotate-certsfrom a VM running on the target Azure Stack stamp can reduce the occurrence of transient issues.
|--api-model||yes||Relative path to the API model (cluster definition) that declares the expected cluster configuration.|
|--ssh-host||yes||Fully qualified domain name (FQDN), or IP address, of an SSH listener that can reach all nodes in the cluster.|
|--linux-ssh-private-key||yes||Path to a valid private SSH key to access the cluster's Linux nodes.|
|--location||yes||Azure location where the cluster is deployed.|
|--subscription-id||yes||Azure subscription where the cluster infra is deployed.|
|--resource-group||yes||Azure resource group where the cluster infra is deployed.|
|--client-id||depends||The service principal client ID. Required if the auth-method is set to client_secret or client_certificate.|
|--client-secret||depends||The service principal client secret. Required if the auth-method is set to client_secret.|
|--azure-env||depends||The target cloud name. Optional if target cloud is AzureCloud.|
|--certificate-profile||no||Relative path to a JSON file containing the new set of certificates.|
|--force||no||Force execution even if the API Server is not responsive.|
Simple steps to rotate certificates
Once you have read all the requirements, run
aks-engine rotate-certs with the appropriate arguments:
./bin/aks-engine rotate-certs \ --location <resource-group-location> \ --api-model <generated-apimodel.json> \ --linux-ssh-private-key <private-SSH-key> \ --ssh-host <apiserver-URI> \ --resource-group <resource-group-name> \ --client-id <service-principal-id> \ --client-secret <service-principal-secret> \ --subscription-id <subscription-id> \ --azure-env <cloud-name>
./bin/aks-engine rotate-certs \ --location "westus2" \ --api-model "_output/my-cluster/apimodel.json" \ --linux-ssh-private-key "~/.ssh/id_rsa" \ --ssh-host "my-cluster.westus2.cloudapp.azure.com"\ --resource-group "my-cluster" \ --client-id "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \ --client-secret "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \ --subscription-id "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \ --azure-env "AzureStackCloud" # optional if targeting AzureCloud
If the certificate rotation process halts before completion due to a failure or transient issue, for example, network connectivity, it is safe to rerun
aks-engine rotate-certs using the
Also notice that
aks-engine rotate-certs logs the output of every step in file
/var/log/azure/rotate-certs.log (Linux) and
For more information on what happens under the hood when running this operation or for further customization, see Under The Hood.
- Read about the The AKS engine on Azure Stack Hub