Pool Configurations

This article explains the configuration options available when you create and edit a pool.

Configure pool

Pool size and auto termination

When you create a pool, in order to control its size, you can set three parameters: minimum idle instances, maxium capacity, and idle instance auto termination.

Minimum Idle Instances

The minimum number of instances the pool keeps idle. These instances do not terminate, regardless of the setting specified in Idle Instance Auto Termination. If a cluster consumes idle instances from the pool, Azure Databricks provisions additional instances to maintain the minimum.

Minimum Idle Instances configuration

Maximum Capacity

The maximum number of instances that the pool will provision. If set, this value constrains all instances (idle + used). If a cluster using the pool requests more instances than this number during autoscaling, the request will fail with an INSTANCE_POOL_MAX_CAPACITY_FAILURE error.

Maximum Capacity configuration

This configuration is optional. Azure Databricks recommend setting a value only in the following circumstances:

  • You have an instance quota you must stay under.
  • You want to protect one set of work from impacting another set of work. For example, suppose your instance quota is 100 and you have teams A and B that need to run jobs. You can create pool A with a max 50 and pool B with max 50 so that the two teams share the 100 quota fairly.
  • You need to cap cost.

Idle Instance Auto Termination

The time in minutes that instances above the value set in Minimum Idle Instances can be idle before being terminated by the pool.

Idle Instance Auto Termination configuration

Instance types

A pool consists of both idle instances kept ready for new clusters and instances in use by running clusters. All of these instances are of the same instance provider type, selected when creating a pool.

A pool’s instance type cannot be edited. Clusters attached to a pool use the same instance type for the driver and worker nodes. Different families of instance types fit different use cases, such as memory-intensive or compute-intensive workloads.

Instance types

Azure Databricks always provides one year’s deprecation notice before ceasing support for an instance type.

Preload Databricks Runtime version

You can speed up cluster launches by selecting a Databricks Runtime version to be loaded on idle instances in the pool. If a user selects that runtime when they create a cluster backed by the pool, that cluster will launch even more quickly than a pool-backed cluster that doesn’t use a preloaded Databricks Runtime version.

Preloaded runtime version

Pool tags

Pool tags allow you to easily monitor the cost of cloud resources used by various groups in your organization. You can specify tags as key-value pairs when you create a pool, and Azure Databricks applies these tags to cloud resources like VMs and disk volumes.

For convenience, Azure Databricks applies three default tags to each pool: Vendor, DatabricksInstancePoolId, and DatabricksInstancePoolCreatorId. You can add custom tags when you create a pool. You can add up to 41 custom tags.

Custom tag inheritance

Pool-backed clusters inherit default and custom tags from the pool configuration. You can also configure cluster-specific custom tags for a pool-backed cluster, and that cluster will apply all custom tags, whether inherited from the pool or assigned to that cluster specifically. You cannot add a cluster-specific custom tag with the same key name as a custom tag inherited from a pool (that is, you cannot override a custom tag that is inherited from the pool).

Custom tag inheritance

Configure custom pool tags

  1. At the bottom of the pool configuration page, select the Tags tab.

  2. Specify a key-value pair for the custom tag.

    Tag key-value pair

  3. Click Add.

Autoscaling local storage

It can often be difficult to estimate how much disk space a particular job will take. To save you from having to estimate how many gigabytes of managed disk to attach to your pool at creation time, Azure Databricks automatically enables autoscaling local storage on all Azure Databricks pools.

With autoscaling local storage, Azure Databricks monitors the amount of free disk space available on your pool’s instances. If an instance runs too low on disk, a new managed disk is attached automatically before it runs out of disk space. Disks are attached up to a limit of 5 TB of total disk space per virtual machine (including the virtual machine’s initial local storage).

The managed disks attached to a virtual machine are detached only when the virtual machine is returned to Azure. That is, managed disks are never detached from a virtual machine as long as it is part of a pool.