HDInsight: - CoC (Calculation of Cores)
If you ever notice “99% of XX HDINSIGHT CORES” in your HDInsight dash board on wonder what it means? Or during cluster creation if you get error/warning “The cluster can have at most 0 data nodes” and wonder what
it means?
In this short article I’ll just explain what above means and show core calculation.
So let’s take an example if you have 740 HDInsight Core on your subscription and 740 HDInsight Core being used by 8 clusters (167 nodes). Each worker node consumes 4 cores and each head node consumes 8 cores.
Here is the calculation:-
Clusters |
Data Nodes x Cores |
Head Nodes x Cores |
Total Cores used |
First cluster |
16 x 4 = 64 |
1 x 8 = 8 |
72 |
Second cluster |
16 x 4 = 64 |
1 x 8 = 8 |
72 |
Third cluster |
16 x 4 = 64 |
1 x 8 = 8 |
72 |
Forth cluster |
16 x 4 =64 |
1 x 8 = 8 |
72 |
Fifth cluster |
32 x 4= 128 |
1 x 8 = 8 |
136 |
Six cluster |
32 x 4= 128 |
1 x 8 = 8 |
136 |
Seven cluster |
32 x 4= 128 |
1 x 8 = 8 |
136 |
Eight cluster |
7 x 4 =28 |
1 x 8 = 8 |
36 |
TOTAL |
668 cores |
64 cores |
732 |
So if you try to create cluster at least with 1 node it requires 12 cores which is not available to subscription. That’s why you get error “The cluster can have at most 0 data nodes” and message “99% of 740 HDInsight cores” in the dashboard.
In case it’s require to increase the core, one of the options is to raise ticket from the portal.
The other option is reuse existing cluster in case you are not using it.
HTH.