question

hanchongtan-0377 avatar image
0 Votes"
hanchongtan-0377 asked hanchongtan-0377 commented

AKS integration with ACI

I have AKS with advanced network (CNI) setup, I'm interested to enable virtual nodes. I understand that if its enabled, then when there's a need to scale the pods beyond the current limit of a node, it will scale to ACI, without the delay of bringing up another node. My question is does it automatically scales the pods in ACI down to 0, after my node has scaled up to accommodate more pods (I have auto scale enabled for my node). How does it work in this situation?

azure-kubernetes-service
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

shivapatpi-MSFT avatar image
2 Votes"
shivapatpi-MSFT answered

Hello @hanchongtan-0377 ,
I was testing your scenario and the answer is NO.
Here is the reason :
When you implement Virtual Nodes and if you want your PODS to be scheduled on those Virtual Nodes - you might have to implement below 2 parts in combination with HPA.
1) Node Affinity
2) Tolerations

Based upon the HPA CPU utilization % it will scale up the PODS , when you have a nodeAffinity like below to reduce the CPU utilization based upon the HPA parameters - If the nodes are not not available , PODS will go and sit on ACI - there by CPU utilization % decreases. So even if a new node gets added as there are no pending PODS + CPU utilization % is less than the target % - hence no new PODS and existing PODS on ACI will not come down to 0.


Example Below YAML:-
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
containers:
- name: php-apache
image: k8s.gcr.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 500m
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- effect: NoSchedule
key: azure.com/aci

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

hanchongtan-0377 avatar image
0 Votes"
hanchongtan-0377 answered

HI @shivapatpi-MSFT , sorry for the delay. I have a long break and forgotten to check on this. To clarify on your suggestion/answer, can I say that it means cluster auto scaling (node scaling) can't be used together with virtual node (ACI)? Because when both is enabled, would it just be adding pods into ACI and not bring up a new node (when max pods per node is hit)?

200203-image.png



image.png (288.8 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

hanchongtan-0377 avatar image
0 Votes"
hanchongtan-0377 answered hanchongtan-0377 commented

Hi @shivapatpi-MSFT , I think I get what you meant. So we can have cluster auto scaling/node scaling enabled, together with virtual node enabled. Then it's up to how we use node affinity & tolerations in the pod yaml configuration to make it scale to the virtual node instead of normal node (non-virtual) when scaling is required. Once its there in the virtual node, it doesn't move it back to a normal node by itself. Is my understanding correct?

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ok, thanks

0 Votes 0 ·