question

TimJin-1583 avatar image
0 Votes"
TimJin-1583 asked TimJin-1583 commented

Add/Delete/Update node pools in AKS thru ARM template

I'm trying to use ARM template to manage the deployment of AKS resources and got following questions thru some experiments by using these attached template:
91541-aks-agentpoolsjson.txt
91542-aks-agentpools-parametersjson.txt
91551-aks-clusterjson.txt
91543-aks-cluster-parametersjson.txt

  1. Changes of node pools in existing cluster template
    I used the attached aks_cluster.json and aks_cluster_parameters.json to create AKS cluster (1 node pool with 2 nodes), then I changed the "count" in aks_cluster_parameters.json to 3 and made the deployment again by using these 2 json files, I can see the deployment was successful and there were 3 nodes in that node pool. Does it mean that configurations of node pool, at least some of them, can be updated in the existing template? Though https://docs.microsoft.com/en-gb/azure/aks/use-multiple-node-pools#manage-node-pools-using-a-resource-manager-template says it only can be done thru a separate template

  2. Adding and deleting node pools in template
    On the AKS cluser created in 1), I added a new agent pool (agentpool2) by using the attached aks_agentpools.json and aks_agentpools_parameters.json, where the configuration of agentpool2 was added as the 2nd item of array. This adding operation was successful. However, when I tried to delete agentpool2 by removing it from that array then making a deployment by using these 2 json files, I can see agentpool2 is still there. Does it mean that ARM template can support adding node pool only but not deleting node pool?

  3. What configuration of node pool can be changed/updated after it is created.
    For the configurations of a node pool listed on https://docs.microsoft.com/en-us/azure/templates/microsoft.containerservice/managedclusters/agentpools?tabs=json#ManagedClusterAgentPoolProfileProperties, which of them can be changed and which of them can be changed thru ARM template, after a node pool is created? I tried to change the vmSize and maxPods thru template but failed. Is it documented somewhere? If no, can I request to document it somewhere?

  4. About "enableAutoScaling"
    I enabled "enableAutoScaling" on an existing node pool with 5 nodes and provided "minCount" as 2 and "maxCount" as 6 accordingly in template. After deployment, 3 nodes were still there which I think is expected.
    Then I disabled it and removed "minCount" and "maxCount" in template. After deployment, I can see "enableAutoScaling" was disabled thru running "az aks nodepool show", but the node count was still 3 though the "count" property of agent pool was 5 in template. It seems to me that "count" property was not respected after disabling "enableAutoScaling", may I know why?


azure-kubernetes-service
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@TimJin-3574 , thanks for being patient, I am working on your questions. I have verified #1 and #4 by myself. #2 is expected behavior, check out How can I delete a resource with ARM template?

For #1, #3 and #4, I have reached out to PG for clarification, I request you to be patient.


0 Votes 0 ·

Hi @HarshitaSingh-MSFT,

Thank you for your investigation. Looking forward to your further response for #1, #3 and #4

For #2, it seems to me that using "Complete" mode is not a workable solution because of "every resource in the target resource group not defined in the template will be deleted" mentioned in that link

We normally have different ARM templates for different resource types in one resource group to get their deployment modularized in a pipeline. For the case of AKS, we also need separate template to manage agent pools. If "Complete" mode is used when deploying agent pool template by using "az deployment group create", I've tested that other resources under same resource group will be deleted. Correct me if I'm wrong.

Is there an workaround for this?

0 Votes 0 ·
singhh-msft avatar image
0 Votes"
singhh-msft answered

@TimJin-3574 , thank you for reaching out to us. After further investigation/discussions on the questions you had asked, I have come up with the responses below:

Does it mean that configurations of node pool, at least some of them, can be updated in the existing template?

I verified again and see that I cannot change any value of the Nodepool through same AKS cluster's ARM template. It throws PropertyChangeNotAllowed error OR does not change anything even after successful update HTTP code. I have made a note of this behavior and given a feedback to the PG. We will work on this issue - thank you for reporting this.


Does it mean that ARM template can support adding node pool only but not deleting node pool?

Yes, you are right. ARM template supports only adding/updating Node Pools/or any other resource for that matter. For deleting any resource, I would suggest you to use PowerShell scripts/CLI. For example, to delete Node Pool, use az aks nodepool delete


For the configurations of a node pool listed on https://docs.microsoft.com/en-us/azure/templates/microsoft.containerservice/managedclusters/agentpools?tabs=json#ManagedClusterAgentPoolProfileProperties, which of them can be changed and which of them can be changed thru ARM template, after a node pool is created? I tried to change the vmSize and maxPods thru template but failed. Is it documented somewhere?

REST, CLI, and ARM all use the same resource provider to perform operations. If you run the following command, you can see the allowed values that can be updated:

 az aks nodepool update --help

I enabled "enableAutoScaling" on an existing node pool with 5 nodes and provided "minCount" as 2 and "maxCount" as 6 accordingly in template. After deployment, 3 nodes were still there which I think is expected.
Then I disabled it and removed "minCount" and "maxCount" in template. After deployment, I can see "enableAutoScaling" was disabled thru running "az aks nodepool show", but the node count was still 3 though the "count" property of agent pool was 5 in template. It seems to me that "count" property was not respected after disabling "enableAutoScaling", may I know why?

I have verified this behavior and it looks unusual and incorrect. I have given a feedback to the PG regarding this and be assured that we will work on this. Again, thank you for reporting this!

For now, please try to use az aks nodepool update for accurate updates in scaling.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TimJin-1583 avatar image
0 Votes"
TimJin-1583 answered TimJin-1583 commented

Hi @HarshitaSingh-MSFT,

Thank you for your effort on investigating these issues.

For #1, I changed the node "count" from 3 to 4 and then changed it back to 3 by using same AKS cluster's ARM template. Both of these 2 operations were successful without any error reported. I also verified the node count after each operation thru portal and "az aks nodepool show" and I can see the node count was changed successfully. My Azure CLI version is 2.22.1 and I run it from Ubuntu WSL. It seems to me that updating node pool thru same AKS cluster's ARM template is not a determinate operation instead of "you can't update an existing Resource Manager template" mentioned on https://docs.microsoft.com/en-gb/azure/aks/use-multiple-node-pools#manage-node-pools-using-a-resource-manager-template, so I do hope Microsoft can make this operation determinate, which will help me build and manage the automatic pipeline of deploying and maintaining AKS

For #4, I used "az aks nodepool update" to disable "enabledAutoScaling" of the agentpool provisioned by ARM template where the node "count" was specified as 5, after the successful run of "az aks nodepool update", I can see the node count was still not respected. Or do you mean that when we need an agentpool which potentially needs "auto scaling" things, we should not provision it thru ARM template but thru Azure CLI "az aks nodepool"? If yes and if it can work, it still brings great complexity/inconvenience when building the pipeline of deploying AKS with multiple agent pools

Appreciate if my following suggestions/requests can be considered/accepted:

  1. Manage AKS with its agent pools in same template instead of keeping original settings of agent pools in one template and updating their settings in another template. In my opinion, this doesn't look good and I don't find why Microsoft does so

  2. If #1 is not achievable, please support deleting agent pools thru that separate ARM template. If no, deleting an agent pool will require a deletion in ARM template and an execution of "az aks nodepool delete" with agent pool name specified as parameter, it makes automatic deployment and management of AKS hard

  3. Document "allowed values that can be updated" of agent pool after its creation somewhere instead of running "az aks nodepool update --help" to get this info. I think this is more friendly to users when maintaining their AKS

  4. Fix the "enableAutoScaling" thing in ARM template asap

· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@TimJin-1583 , thank you for your questions.

For #1(original question):
Yes certain things can't be changed, and we need to update the articles to be more explicit (thanks for helping us improve on that): The first node pool is a system pool, and you should create additional node pools in user mode for production systems and your application pods. See our article that discusses multiple node pools here and we discuss system pool here.

For #2:
Acknowledge that. Pls create a Feature request here.



0 Votes 0 ·

@HarshitaSingh-MSFT I have created the requests on https://feedback.azure.com/forums/281804-azure-resource-manager/suggestions/43369284-support-provisioning-updating-deleting-agent-pools

Yeah, fully agree with you to create user mode agent pools for my application pods. I'm building our development environment so the setup of AKS with agent pool in this dev environment is simple. In the production environment it will have multiple agent pools within one AKS cluster and that's why I DO HOPE Microsoft to provide support of fully managing AKS agent pools in one ARM template. So I'm wondering if I can get update on whether my requests can be accepted and if yes when it can be done

1 Vote 1 ·

For #3:
Acknowledge that. We will make changes soon.

For #4:
Request you to take a look at this section:

94455-microsoftteams-image-13.png

Let me know if this answers your question.

0 Votes 0 ·

@HarshitaSingh-MSFT , no it doesn't answer my question. The issue is that after disabling "enableAutoScaling" of agent pool, the "count" property of that agent pool is not respected.

I saw this issue from the following work I did:

"I enabled "enableAutoScaling" on an existing node pool with 5 nodes and provided "minCount" as 2 and "maxCount" as 6 accordingly in template. After deployment, 3 nodes were still there which I think is expected.
Then I disabled it and removed "minCount" and "maxCount" in template. After deployment, I can see "enableAutoScaling" was disabled thru running "az aks nodepool show", but the node count was still 3 though the "count" property of agent pool was 5 in template."

I tried to disable "enableAutoScaling" by both ARM template and Azure CLI, the result is same

0 Votes 0 ·

@TimJin-1583, this looks like an issue - we have made a note of this and be assured that we are working on it. If the posted answer thread helps you in addressing your queries, please mark it as an answer as it will help us and the community.

0 Votes 0 ·
Show more comments