群集 APIClusters API

• 12/01/2020
• • 
  • 

利用群集 API，可以创建、启动、编辑、列出、终止和删除群集。The Clusters API allows you to create, start, edit, list, terminate, and delete clusters. 对群集 API 的请求的最大允许大小为 10MB。The maximum allowed size of a request to the Clusters API is 10MB.

群集生命周期方法需要从创建返回的群集 ID。Cluster lifecycle methods require a cluster ID, which is returned from Create. 若要获取群集列表，请调用列表。To obtain a list of clusters, invoke List.

Azure Databricks 将群集节点实例类型映射到被称为 DBU 的计算单位。Azure Databricks maps cluster node instance types to compute units known as DBUs. 有关受支持的实例类型及其对应 DBU 的列表，请参阅实例类型定价页。See the instance type pricing page for a list of the supported instance types and their corresponding DBUs. 有关实例提供程序的信息，请参阅 Azure 实例类型规范和定价。For instance provider information, see Azure instance type specifications and pricing.

Azure Databricks 在停止支持实例类型之前，始终会提供为期一年的弃用通知。Azure Databricks always provides one year’s deprecation notice before ceasing support for an instance type.

创建 Create

创建新的 Apache Spark 群集。Create a new Apache Spark cluster. 如果有必要，此方法会从云服务提供商处获取新实例。This method acquires new instances from the cloud provider if necessary. 此方法是异步的；返回的 cluster_id 可用于轮询群集状态。This method is asynchronous; the returned cluster_id can be used to poll the cluster state. 在此方法返回时，群集处于 PENDING 状态。When this method returns, the cluster is in a PENDING state. 群集在进入 RUNNING 状态后即可供使用。The cluster is usable once it enters a RUNNING state. 请参阅 ClusterState。See ClusterState.

示例请求：An example request:

{
  "cluster_name": "my-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "spark_conf": {
    "spark.speculation": true
  },
  "num_workers": 25
}

下面是一个自动缩放的群集的示例。Here is an example for an autoscaling cluster. 此群集最初将包含 2 个节点（最小值）。This cluster will start with 2 nodes, the minimum.

{
  "cluster_name": "autoscaling-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "autoscale" : {
    "min_workers": 2,
    "max_workers": 50
  }
}

下面是包含要使用的策略的示例。Here is an example that includes the policy to use.

{
    "num_workers": null,
    "autoscale": {
        "min_workers": 2,
        "max_workers": 8
    },
    "cluster_name": "my-cluster",
    "spark_version": "7.3.x-scala2.12",
    "spark_conf": {},
    "node_type_id": "Standard_D3_v2",
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "init_scripts": [],
    "policy_id": "C65B864F02000008"
}

通过该 API 创建使用策略的作业Create a job using a policy with the API

若要使用策略通过新群集来创建作业或提交运行，请将 policy_id 属性添加到请求的 new_cluster 规范。To create a job or submit a run with a new cluster using a policy, add the policy_id attribute to the request’s new_cluster specification.

{
  "run_name": "my spark task",
  "new_cluster": {
    "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10,
    "policy_id": "ABCD000000000000"
  },
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

请求结构 Request structure

响应结构 Response structure

编辑 Edit

编辑群集的配置，以匹配所提供的属性和大小。Edit the configuration of a cluster to match the provided attributes and size.

如果群集处于 RUNNING 或 TERMINATED 状态，则可编辑该群集。You can edit a cluster if it is in a RUNNING or TERMINATED state. 如果在群集处于 RUNNING 状态时编辑该群集，它将会重启，以便使新属性能够生效。If you edit a cluster while it is in a RUNNING state, it will be restarted so that the new attributes can take effect. 如果在群集处于 TERMINATED 状态时编辑该群集，它将保持 TERMINATED 状态。If you edit a cluster while it is in a TERMINATED state, it will remain TERMINATED. 在下次使用 clusters/start API 启动它时，新属性将会生效。The next time it is started using the clusters/start API, the new attributes will take effect. 尝试编辑处于任何其他状态的群集将会被拒绝，并返回 INVALID_STATE 错误代码。An attempt to edit a cluster in any other state will be rejected with an INVALID_STATE error code.

无法编辑由 Databricks 作业服务创建的群集。Clusters created by the Databricks Jobs service cannot be edited.

示例请求：An example request:

{
 "cluster_id": "1202-211320-brick1",
 "num_workers": 10,
 "spark_version": "7.3.x-scala2.12",
 "node_type_id": "Standard_D3_v2"
}

请求结构 Request structure

启动 Start

启动已终止的群集（如果提供了该群集的 ID）。Start a terminated cluster given its ID. 这类似于 createCluster，区别在于：This is similar to createCluster, except:

• 已终止的群集 ID 和属性会被保留。The terminated cluster ID and attributes are preserved.
• 该群集启动时的大小为最后指定的群集大小。The cluster starts with the last specified cluster size. 如果已终止的群集是自动缩放的群集，该群集启动时只包含最少数量的节点。If the terminated cluster is an autoscaling cluster, the cluster starts with the minimum number of nodes.
• 如果该群集处于 RESTARTING 状态，则会返回 400 错误。If the cluster is in the RESTARTING state, a 400 error is returned.
• 你无法启动为运行作业而启动的群集。You cannot start a cluster launched to run a job.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

重启 Restart

如果有群集的 ID，则重启该群集。Restart a cluster given its ID. 该群集必须处于 RUNNING 状态。The cluster must be in the RUNNING state.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

重设大小 Resize

重设群集大小，使其具有所需的工作器数。Resize a cluster to have a desired number of workers. 该群集必须处于 RUNNING 状态。The cluster must be in the RUNNING state.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1",
  "num_workers": 30
}

请求结构 Request structure

删除（终止） Delete (terminate)

终止群集（如果提供了该群集的 ID）。Terminate a cluster given its ID. 该群集会以异步方式被删除。The cluster is removed asynchronously. 终止完成之后，该群集将会处于 TERMINATED 状态。Once the termination has completed, the cluster will be in the TERMINATED state. 如果该群集已处于 TERMINATING 或 TERMINATED 状态，则不会执行任何操作。If the cluster is already in a TERMINATING or TERMINATED state, nothing will happen.

群集在终止 30 天后会永久删除。30 days after a cluster is terminated, it is permanently deleted.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

永久删除 Permanent delete

永久删除群集。Permanently delete a cluster. 如果该群集正在运行，则它会被终止，并且其资源会以异步方式删除。If the cluster is running, it is terminated and its resources are asynchronously removed. 如果该群集已终止，则它会被立即删除。If the cluster is terminated, then it is immediately removed.

对于永久删除的群集，无法执行任何操作，包括检索该群集的权限。You cannot perform any action, including retrieve the cluster’s permissions, on a permanently deleted cluster. 群集列表中也不会再返回已永久删除的群集。A permanently deleted cluster is also no longer returned in the cluster list.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

获取 Get

检索群集的信息（如果提供了该群集的标识符）。Retrieve the information for a cluster given its identifier. 当群集正在运行时，或在群集被终止之后的 30 天之内，可以描述群集。Clusters can be described while they are running or up to 30 days after they are terminated.

示例请求：An example request:

/clusters/get?cluster_id=1202-211320-brick1

请求结构 Request structure

响应结构 Response structure

固定 Pin

确保即使在群集已被终止的时间超过了 30 天的情况下仍会保留通用群集配置。Ensure that an all-purpose cluster configuration is retained even after a cluster has been terminated for more than 30 days. “固定”可确保群集始终会由列表 API 返回。Pinning ensures that the cluster is always returned by the List API. 固定一个已经固定的群集不会起作用。Pinning a cluster that is already pinned has no effect.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

取消固定 Unpin

允许最终从列表 API 返回的列表中删除群集。Allows the cluster to eventually be removed from the list returned by the List API. 对一个未固定的群集取消固定不会起作用。Unpinning a cluster that is not pinned has no effect.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

列表 List

返回有关所有固定群集、活动群集、过去 30 天内最多 70 个最近终止的通用群集以及过去 30 天内最多 30 个最近终止的作业群集的相关信息。Return information about all pinned clusters, active clusters, up to 70 of the most recently terminated all-purpose clusters in the past 30 days, and up to 30 of the most recently terminated job clusters in the past 30 days. 例如，如果有 1 个固定群集、4 个活动群集、45 个在过去 30 天内终止的通用群集以及 50 个在过去 30 天内终止的作业群集，则此 API 会返回 1 个固定群集、4 个活动群集、所有 45 个终止的通用群集，以及 30 个最近终止的作业群集。For example, if there is 1 pinned cluster, 4 active clusters, 45 terminated all-purpose clusters in the past 30 days, and 50 terminated job clusters in the past 30 days, then this API returns the 1 pinned cluster, 4 active clusters, all 45 terminated all-purpose clusters, and the 30 most recently terminated job clusters.

响应结构 Response structure

列出节点类型 List node types

返回受支持的 Spark 节点类型的列表。Return a list of supported Spark node types. 这些节点类型可用于启动群集。These node types can be used to launch a cluster.

响应结构 Response structure

运行时版本 Runtime versions

返回可用的运行时版本的列表。Return the list of available runtime versions. 这些版本可用于启动群集。These versions can be used to launch a cluster.

响应结构 Response structure

事件 Events

检索有关群集活动性的事件列表。Retrieve a list of events about the activity of a cluster. 你可以检索活动的（正在运行的、挂起的或正在重新配置的）群集和已终止的群集（自其上次终止后 30 天内）中的事件。You can retrieve events from active clusters (running, pending, or reconfiguring) and terminated clusters within 30 days of their last termination. 此 API 支持分页。This API is paginated. 如果有更多事件需要读取，响应会包括用于请求下一页事件所需的所有参数。If there are more events to read, the response includes all the parameters necessary to request the next page of events.

示例请求：An example request:

{
  "cluster_id": "1202-211320-brick1"
}

示例响应：An example response:

{
  "events": [{
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1534371918659,
    "type": "TERMINATING",
    "details": {
      "reason": {
        "code": "INACTIVITY",
        "parameters": {
          "inactivity_duration_min": "120"
        }
      }
    }
  }, {
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1534358289590,
    "type": "RUNNING",
    "details": {
      "current_num_workers": 2,
      "target_num_workers": 2
    }
  }, {
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1533225298406,
    "type": "RESTARTING",
    "details": {
      "user": "admin"
    }
  }],
  "next_page": {
    "cluster_id": "0802-034608-aloe926",
    "end_time": 1534371918659,
    "offset": 50
  },
  "total_count": 55
}

用于检索下一页事件的请求示例：Example request to retrieve the next page of events:

{
  "cluster_id": "1202-211320",
  "start_time": 1534371918659
}

请求结构 Request structure

检索与某个特定群集相关的事件。Retrieve events pertaining to a specific cluster.

响应结构 Response structure

数据结构 Data structures

本节内容：In this section:

• 自动缩放AutoScale
• ClusterInfoClusterInfo
• ClusterEventClusterEvent
• ClusterEventTypeClusterEventType
• EventDetailsEventDetails
• ClusterAttributesClusterAttributes
• clusterSizeClusterSize
• ListOrderListOrder
• ResizeCauseResizeCause
• ClusterLogConfClusterLogConf
• InitScriptInfoInitScriptInfo
• ClusterTagClusterTag
• DbfsStorageInfoDbfsStorageInfo
• FileStorageInfoFileStorageInfo
• DockerImageDockerImage
• DockerBasicAuthDockerBasicAuth
• LogSyncStatusLogSyncStatus
• NodeTypeNodeType
• ClusterCloudProviderNodeInfoClusterCloudProviderNodeInfo
• ClusterCloudProviderNodeStatusClusterCloudProviderNodeStatus
• ParameterPairParameterPair
• SparkConfPairSparkConfPair
• SparkEnvPairSparkEnvPair
• SparkNodeSparkNode
• SparkVersionSparkVersion
• TerminationReasonTerminationReason
• PoolClusterTerminationCodePoolClusterTerminationCode
• ClusterSourceClusterSource
• ClusterStateClusterState
• TerminationCodeTerminationCode
• TerminationTypeTerminationType
• TerminationParameterTerminationParameter

AutoScale AutoScale

定义群集工作器的最小数量和最大数量的范围。Range defining the min and max number of cluster workers.

ClusterInfo ClusterInfo

有关群集的元数据。Metadata about a cluster.

ClusterEvent ClusterEvent

群集事件信息。Cluster event information.

ClusterEventType ClusterEventType

群集事件的类型。Type of a cluster event.

EventDetails EventDetails

有关群集事件的详细信息。Details about a cluster event.

ClusterAttributes ClusterAttributes

在群集创建过程中设置的一组常见属性。Common set of attributes set during cluster creation. 在群集的生存期内无法更改这些属性。These attributes cannot be changed over the lifetime of a cluster.

ClusterSize ClusterSize

群集大小规范。Cluster size specification.

ListOrder ListOrder

基于列表的查询的泛型排序枚举。Generic ordering enum for list-based queries.

ResizeCause ResizeCause

重设群集大小的原因。Reason why a cluster was resized.

ClusterLogConf ClusterLogConf

群集日志的路径。Path to cluster log.

InitScriptInfo InitScriptInfo

初始化脚本的路径。Path to an init script. 若要了解如何将初始化脚本与 Databricks 容器服务配合使用，请参阅使用初始化脚本。For instructions on using init scripts with Databricks Container Services, see Use an init script.

ClusterTag ClusterTag

群集标记定义。Cluster tag definition.

DbfsStorageInfo DbfsStorageInfo

DBFS 存储信息。DBFS storage information.

FileStorageInfo FileStorageInfo

文件存储信息。File storage information.

DockerImageDockerImage

Docker 映像连接信息。Docker image connection information.

DockerBasicAuthDockerBasicAuth

Docker 存储库基本身份验证信息。Docker repository basic authentication information.

LogSyncStatus LogSyncStatus

日志传送状态。Log delivery status.

NodeType NodeType

Spark 节点类型的说明，包括节点的维度以及将会在其上托管该节点的实例类型。Description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

ClusterCloudProviderNodeInfoClusterCloudProviderNodeInfo

有关云服务提供商所提供实例的信息。Information about an instance supplied by a cloud provider.

ClusterCloudProviderNodeStatusClusterCloudProviderNodeStatus

云服务提供商所提供实例的状态。Status of an instance supplied by a cloud provider.

ParameterPair ParameterPair

提供有关群集终止原因的更多信息的参数。Parameter that provides additional information about why a cluster was terminated.

SparkConfPair SparkConfPair

Spark 配置键值对。Spark configuration key-value pairs.

SparkEnvPair SparkEnvPair

Spark 环境变量键值对。Spark environment variable key-value pairs.

SparkNode SparkNode

Spark 驱动程序或执行程序配置。Spark driver or executor configuration.

SparkVersion SparkVersion

群集的 Databricks Runtime 版本。Databricks Runtime version of the cluster.

TerminationReason TerminationReason

终止群集的原因。Reason why a cluster was terminated.

PoolClusterTerminationCode PoolClusterTerminationCode

状态代码，用于指示群集因池故障而终止的原因。Status code indicating why the cluster was terminated due to a pool failure.

ClusterSource ClusterSource

创建该群集的服务。Service that created the cluster.

ClusterState ClusterState

群集的状态。State of a cluster. 可允许的状态转换如下：The allowable state transitions are as follows:

• PENDING -> RUNNING
• PENDING -> TERMINATING
• RUNNING -> RESIZING
• RUNNING -> RESTARTING
• RUNNING -> TERMINATING
• RESTARTING -> RUNNING
• RESTARTING -> TERMINATING
• RESIZING -> RUNNING
• RESIZING -> TERMINATING
• TERMINATING -> TERMINATED

TerminationCode TerminationCode

指示该群集终止原因的状态代码。Status code indicating why the cluster was terminated.

TerminationType TerminationType

群集的终止原因。Reason why the cluster was terminated.

TerminationParameter TerminationParameter

提供群集终止原因其他相关信息的键。Key that provides additional information about why a cluster was terminated.