Capacity policy

A capacity policy is used for controlling the compute resources of data management operations on the cluster.

The capacity policy object

The capacity policy is made of:

Ingestion capacity

Property Type Description
ClusterMaximumConcurrentOperations long A maximal value for the number of concurrent ingestion operations in a cluster.
CoreUtilizationCoefficient double A coefficient for the percentage of cores to use when calculating the ingestion capacity. The calculation's result will always be normalized by ClusterMaximumConcurrentOperations
The cluster's total ingestion capacity, as shown by .show capacity, is calculated by:
Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

Note

In clusters with three or more nodes, the admin node doesn't participate in ingestion operations. The Number of nodes in cluster is reduced by one.

Extents merge capacity

Property Type Description
MinimumConcurrentOperationsPerNode long A minimal value for the number of concurrent extents merge/rebuild operations on a single node. Default is 1
MaximumConcurrentOperationsPerNode long A maximal value for the number of concurrent extents merge/rebuild operations on a single node. Default is 3

The cluster's total extents merge capacity, as shown by .show capacity, is calculated by:

Number of nodes in cluster x Concurrent operations per node

The effective value for Concurrent operations per node gets automatically adjusted by the system in the range [MinimumConcurrentOperationsPerNode,MaximumConcurrentOperationsPerNode], as long as the success rate of the merge operations is 90% or higher.

Note

In clusters with three or more nodes, the admin node doesn't participate in doing merge operations. The Number of nodes in cluster is reduced by one.

Extents purge rebuild capacity

Property Type Description
MaximumConcurrentOperationsPerNode long A maximal value for the number of concurrent rebuild extents for purge operations on a single node

The cluster's total extents purge rebuild capacity (as shown by .show capacity) is calculated by:

Number of nodes in cluster x MaximumConcurrentOperationsPerNode

Note

In clusters with three or more nodes, the admin node doesn't participate in doing merge operations. The Number of nodes in cluster is reduced by one.

Export capacity

Property Type Description
ClusterMaximumConcurrentOperations long A maximal value for the number of concurrent export operations in a cluster.
CoreUtilizationCoefficient double A coefficient for the percentage of cores to use when calculating the export capacity. The calculation's result will always be normalized by ClusterMaximumConcurrentOperations.

The cluster's total export capacity, as shown by .show capacity, is calculated by:

Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

Note

In clusters with three or more nodes, the admin node doesn't participate in export operations. The Number of nodes in cluster is reduced by one.

Extents partition capacity

Property Type Description
ClusterMinimumConcurrentOperations long A minimal value for the number of concurrent extents partition operations in a cluster. Default: 1
ClusterMaximumConcurrentOperations long A maximal value for the number of concurrent extents partition operations in a cluster. Default: 32

The cluster's total extents partition capacity (as shown by .show capacity).

The effective value for Concurrent operations is automatically adjusted by the system in the range [ClusterMinimumConcurrentOperations,ClusterMaximumConcurrentOperations], as long as the success rate of the partitioning operations is 90% or higher.

Materialized views capacity policy

The policy can be used to change concurrency settings for materialized views. This change may be required when there's more than a single materialized view defined on a cluster, and the cluster can't keep up with the materialization of all views. By default, concurrency settings are relatively low to ensure that materialization doesn't impact cluster's performance.

Warning

The materialized view capacity policy should only be increased if the cluster's resources are well (low CPU, available memory). Increasing these values when resources are limited may result in resources exhaustion and will badly impact the cluster's performance.

The materialized views capacity policy is part of the cluster's capacity policy, and has the following JSON representation:

{
   "MaterializedViewsCapacity": {
    "ClusterMaximumConcurrentOperations": 1,
    "ExtentsRebuildCapacity": {
      "ClusterMaximumConcurrentOperations": 50,
      "MaximumConcurrentOperationsPerNode": 5
    }
  }
}

Properties

Property Description
ClusterMaximumConcurrentOperations The maximum number of materialized views that the cluster can materialize concurrently. This value is 1 by default, while materialization itself (of a single individual view) may run many concurrent operations. If there's more than a single materialized view defined on the cluster, and if the cluster's resources are in good state, it's recommended to increase this value.
ExtentsRebuildCapacity Determines the number of concurrent extents rebuild operations, executed for all materialized views during the materialization process. If several views are executing concurrently, since ClusterMaximumConcurrentOperation is greater than 1, they'll share the quota defined by this property. The maximum number of concurrent extents rebuild operations won't exceed this value.

Extents rebuild capacity

For more information about extents rebuild operations, see how materialized views work. This setting is only relevant to Engine V2 clusters. This setting is not relevant to EngineV3 clusters.

The maximum number of extents rebuild is calculated by:

Maximum(`ClusterMaximumConcurrentOperations`, `Number of nodes in cluster` * `MaximumConcurrentOperationsPerNode`)
  • Default values are 50 total concurrency rebuilds and maximum 5 per node.
  • The ExtentsRebuildCapacity policy serves as an upper limit only. The actual value used is dynamically determined by the system, based on current cluster's conditions (memory, CPU) and an estimation of the amount of resources required by the rebuild operation. In practice, concurrency can be much lower than the value specified in capacity policy.
    • The MaterializedViewExtentsRebuild metric provides information about how many extents were rebuilt in each materialization cycle. For more information, see materialized views monitoring.

Stored query results capacity

Property Type Description
MaximumConcurrentOperationsPerDbAdmin long The maximum number of concurrent ingestion operations in a cluster admin node.
CoreUtilizationCoefficient double A coefficient for the percentage of cores to use when calculating the stored query results creation capacity.

The cluster's total stored query results creation capacity, as shown by .show capacity, is calculated by:

Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient)

Defaults

The default capacity policy has the following JSON representation:

{
  "IngestionCapacity": {
    "ClusterMaximumConcurrentOperations": 512,
    "CoreUtilizationCoefficient": 0.75
  },
  "ExtentsMergeCapacity": {
    "MinimumConcurrentOperationsPerNode": 1,
    "MaximumConcurrentOperationsPerNode": 3
  },
  "ExtentsPurgeRebuildCapacity": {
    "MaximumConcurrentOperationsPerNode": 1
  },
  "ExportCapacity": {
    "ClusterMaximumConcurrentOperations": 100,
    "CoreUtilizationCoefficient": 0.25
  },
  "ExtentsPartitionCapacity": {
    "ClusterMinimumConcurrentOperations": 1,
    "ClusterMaximumConcurrentOperations": 32
  },
  "MaterializedViewsCapacity": {
    "ClusterMaximumConcurrentOperations": 1,
    "ExtentsRebuildCapacity": {
      "ClusterMaximumConcurrentOperations": 50,
      "MaximumConcurrentOperationsPerNode": 5
    }
  },
  "StoredQueryResultsCapacity": {
    "MaximumConcurrentOperationsPerDbAdmin": 250,
    "CoreUtilizationCoefficient": 0.75
  }
}

Control commands

Warning

Consult with the Azure Data Explorer team before altering a capacity policy.

Control commands throttling

Kusto limits the number of concurrent requests for the following user-initiated commands:

  • Ingestions
  • Purges
    • The global limit is currently fixed at one per cluster.
    • The purge rebuild capacity is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands won't be blocked or throttled because of this process, but will complete faster or slower depending on the purge rebuild capacity.
  • Exports

When the cluster detects that an operation has exceeded the limit on concurrent requests:

  • The command's state, as presented by System information commands, will be Throttled.
  • The error message will include the command type, the origin of the throttling and the capacity that's been exceeded. For example:
    • For example: The control command was aborted due to throttling. Retrying after some backoff might succeed. CommandType: 'TableSetOrAppend', Capacity: 18, Origin: 'CapacityPolicy/Ingestion'.
  • The HTTP response code will be 429. The subcode will be TooManyRequests.
  • The exception type will be ControlCommandThrottledException.

Note

Control commands may also be throttled as a result of exceeding the limit defined by a workload group's Request rate limit policy.