Is Rotating The Service Principal Without Restarting AKS Nodes Possible?

Question

When following the steps in the Azure Kubernetes Service documentation to update/rotate the Service Principal, one of the steps carried out by the CLI utility involves restarting each Node in the cluster in a rolling fashion, I think twice per node. This caught me out by surprise the first time I rotated the keys on a cluster.

We use AKS to run a platform which relies on persistent TCP connections being formed from customer sites to AKS. If the node running that particular pod reboots (or indeed that pod restarts), the customer sites would automatically (re) connect to another node, but with the side effect of closing all sessions that customer might have open.

Completely appreciate this may not be possible but I'm wondering if there is a method to rotate the Service Principal in AKS without rebooting? Cheers!

Answer 1

Hello @Jamie W ,
Thanks for reaching out to Microsoft Q&A Platform. As of today every service principal update will require a reboot of each worker nodes. When we update the AKS cluster with the new SP credentials at first it will update the Master Node with new client_secret. Then it will go to each node and will update the config /etc/kubernetes/azure.json with the new secret. You can also manually login to each node and update those attributes but not a best practice because the Master Node will still use the old password which might lead to other consequences like any scale up will come up old SP credentials.

azure.json config file looks like below

Considering your customer scenario , probably you can renew the service principal for the extended period of time (for example 10 years ) and do the update only once.
az ad sp credential reset --name $SPID --years 10

Kindly let us know if you have additional questions.

Regards,
Shiva.