Rotate Kubernetes certificates on Azure Stack Hub

This document provides guidance on how to rotate certificates on an existing AKS Engine cluster and recommendations for using adopting aks-engine rotate-certs as a tool.

Important

This feature is currently in public preview. This preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

This guide assumes that you already have deployed a cluster using AKS engine and the cluster is in a healthy state.

Planning for certificate rotation

When considering using this functionality, be aware that the Kubernetes control plane will be unavailable during the update, validation, and restart steps. Plan this maintenance operation accordingly. Also, plan to execute this operation in a staging environment with equal configuration to the production environment before trying in production.

Review the following considerations before attempting this operation:

  • You will need access to the API model (apimodel.json) that was generated by the commands aks-engine deploy or aks-engine generate. By default this file is placed into a relative directory such as _output/<clustername>/.

  • An aks-engine rotate-certs operation causes API server downtime.

  • aks-engine rotate-certs expects an API model that conforms to the current state of the cluster. aks-engine rotate-certs executes remote commands on the cluster nodes and uses the API model information to establish a secure SSH connection. aks-engine rotate-certs also relies on some resources to be named in accordance with the original aks-engine deployment, for example, VMs must follow the naming provided by aks-engine.

  • aks-engine rotate-certs relies upon a working connection to the cluster control plane during certificate rotation:

    • To validate each step of the process.
    • To restart/recreate cluster resources such as kube-system pods and service account tokens.

    If you are rotating the certificates of a cluster in a VNet closed to outside access, you must run aks-engine rotate-certs from a host VM that has network access to the control plane, for example, a jumpbox VM that resides in the same VNet as the master VMs.

  • If you are using aks-engine rotate-certs in production, it is recommended to stage a certificate rotation test on a cluster that was built to the same specifications. That is, the cluster is built with the same cluster configuration, the same version of the AKS engine command-line tool, and the same set of enabled addons as your production cluster before performing the certificate rotation. The AKS engine supports different cluster configurations and the extent of end-to-end testing that the AKS engine team runs cannot practically cover every possible configuration. Therefore, it is recommended that you ensure in a staging environment that your specific cluster configuration works with aks-engine rotate-certs before attempting the operation on your production cluster.

  • aks-engine rotate-certs does not guarantee backwards compatibility. If you deployed with aks-engine version 0.60.x, you should prefer executing the certificate rotation process with version 0.60.x.

  • Fetching a new set of certificates from Key Vault is not supported at this point.

  • Use a reliable network connection. aks-engine rotate-certs requires the execution of multiple remote commands, which are subject to potential failures, mostly if the connection to the cluster nodes is not reliable. Running aks-engine rotate-certs from a VM running on the target Azure Stack stamp can reduce the occurrence of transient issues.

Parameters

Parameter Required Description
--api-model yes Relative path to the API model (cluster definition) that declares the expected cluster configuration.
--ssh-host yes Fully qualified domain name (FQDN), or IP address, of an SSH listener that can reach all nodes in the cluster.
--linux-ssh-private-key yes Path to a valid private SSH key to access the cluster's Linux nodes.
--location yes Azure location where the cluster is deployed.
--subscription-id yes Azure subscription where the cluster infra is deployed.
--resource-group yes Azure resource group where the cluster infra is deployed.
--client-id depends The service principal client ID. Required if the auth-method is set to client_secret or client_certificate.
--client-secret depends The service principal client secret. Required if the auth-method is set to client_secret.
--azure-env depends The target cloud name. Optional if target cloud is AzureCloud.
--certificate-profile no Relative path to a JSON file containing the new set of certificates.
--force no Force execution even if the API Server is not responsive.

Simple steps to rotate certificates

Once you have read all the requirements, run aks-engine rotate-certs with the appropriate arguments:

./bin/aks-engine rotate-certs \
  --location <resource-group-location> \
  --api-model <generated-apimodel.json> \
  --linux-ssh-private-key <private-SSH-key> \
  --ssh-host <apiserver-URI> \
  --resource-group <resource-group-name> \
  --client-id <service-principal-id> \
  --client-secret <service-principal-secret> \
  --subscription-id <subscription-id> \
  --azure-env <cloud-name>

For example:

./bin/aks-engine rotate-certs \
  --location "westus2" \
  --api-model "_output/my-cluster/apimodel.json" \
  --linux-ssh-private-key "~/.ssh/id_rsa" \
  --ssh-host "my-cluster.westus2.cloudapp.azure.com"\
  --resource-group "my-cluster" \
  --client-id "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \
  --client-secret "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \
  --subscription-id "12345678-XXXX-YYYY-ZZZZ-1234567890ab" \
  --azure-env "AzureStackCloud" # optional if targeting AzureCloud

Troubleshooting

If the certificate rotation process halts before completion due to a failure or transient issue, for example, network connectivity, it is safe to rerun aks-engine rotate-certs using the --force flag.

Also notice that aks-engine rotate-certs logs the output of every step in file /var/log/azure/rotate-certs.log (Linux) and c:\\k\\rotate-certs.log (Windows).

For more information on what happens under the hood when running this operation or for further customization, see Under The Hood.

Next steps