群集 APIClusters API

利用群集 API,可以创建、启动、编辑、列出、终止和删除群集。The Clusters API allows you to create, start, edit, list, terminate, and delete clusters. 对群集 API 的请求的最大允许大小为 10MB。The maximum allowed size of a request to the Clusters API is 10MB.

群集生命周期方法需要从创建返回的群集 ID。Cluster lifecycle methods require a cluster ID, which is returned from Create. 若要获取群集列表,请调用列表To obtain a list of clusters, invoke List.

Azure Databricks 将群集节点实例类型映射到被称为 DBU 的计算单位。Azure Databricks maps cluster node instance types to compute units known as DBUs. 有关受支持的实例类型及其对应 DBU 的列表,请参阅实例类型定价页See the instance type pricing page for a list of the supported instance types and their corresponding DBUs. 有关实例提供程序的信息,请参阅 Azure 实例类型规范和定价For instance provider information, see Azure instance type specifications and pricing.

Azure Databricks 在停止支持实例类型之前,始终会提供为期一年的弃用通知。Azure Databricks always provides one year’s deprecation notice before ceasing support for an instance type.

重要

要访问 Databricks REST API,必须进行身份验证To access Databricks REST APIs, you must authenticate.

创建 Create

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/create POST

创建新的 Apache Spark 群集。Create a new Apache Spark cluster. 如果有必要,此方法会从云服务提供商处获取新实例。This method acquires new instances from the cloud provider if necessary. 此方法是异步的;返回的 cluster_id 可用于轮询群集状态。This method is asynchronous; the returned cluster_id can be used to poll the cluster state. 在此方法返回时,群集处于 PENDING 状态。When this method returns, the cluster is in a PENDING state. 群集在进入 RUNNING 状态后即可供使用。The cluster is usable once it enters a RUNNING state. 请参阅 ClusterStateSee ClusterState.

备注

由于云服务提供商的限制或暂时性的网络问题,Azure Databricks 可能无法获取某些已请求的节点。Azure Databricks may not be able to acquire some of the requested nodes, due to cloud provider limitations or transient network issues. 如果无法获取足够数量的已请求节点,群集创建将会终止,并显示一条信息性的错误消息。If it is unable to acquire a sufficient number of the requested nodes, cluster creation will terminate with an informative error message.

示例请求:An example request:

{
  "cluster_name": "my-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "spark_conf": {
    "spark.speculation": true
  },
  "num_workers": 25
}

下面是一个自动缩放的群集的示例。Here is an example for an autoscaling cluster. 此群集最初将包含 2 个节点(最小值)。This cluster will start with 2 nodes, the minimum.

{
  "cluster_name": "autoscaling-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "autoscale" : {
    "min_workers": 2,
    "max_workers": 50
  }
}

下面是包含要使用的策略的示例。Here is an example that includes the policy to use.

{
    "num_workers": null,
    "autoscale": {
        "min_workers": 2,
        "max_workers": 8
    },
    "cluster_name": "my-cluster",
    "spark_version": "7.3.x-scala2.12",
    "spark_conf": {},
    "node_type_id": "Standard_D3_v2",
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "init_scripts": [],
    "policy_id": "C65B864F02000008"
}

通过该 API 创建使用策略的作业Create a job using a policy with the API

若要使用策略通过新群集来创建作业或提交运行,请将 policy_id 属性添加到请求的 new_cluster 规范。To create a job or submit a run with a new cluster using a policy, add the policy_id attribute to the request’s new_cluster specification.

{
  "run_name": "my spark task",
  "new_cluster": {
    "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10,
    "policy_id": "ABCD000000000000"
  },
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

注意: 在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段将会立即更新,以反映 10 个工作器的目标大小,而 executors 中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_namecluster_name STRING 用户请求的群集名称。Cluster name requested by the user. 此名称不必唯一。This doesn’t have to be unique. 如果在创建时未指定此字段,群集名称将为空字符串。If not specified at creation, the cluster name will be an empty string.
spark_versionspark_version STRING 群集的运行时版本。The runtime version of the cluster. 可以通过使用运行时版本 API 调用来检索可用的运行时版本的列表。You can retrieve a list of available runtime versions by using the Runtime versions API call. 此字段为必需字段。This field is required.
spark_confspark_conf SparkConfPairSparkConfPair 一个对象,其中包含一组可选的由用户指定的 Spark 配置键值对。An object containing a set of optional, user-specified Spark configuration key-value pairs. 你也可以分别通过以下属性,将额外 JVM 选项的字符串传入到驱动程序和执行程序:You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsspark.executor.extraJavaOptionsspark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

示例 Spark 配置:Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5}{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_idnode_type_id STRING 此字段通过单个值对提供给此群集中的每个 Spark 节点的资源进行编码。This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. 例如,可以针对内存密集型或计算密集型的工作负载来预配和优化 Spark 节点。通过使用列出节点类型 API 调用可以检索可用节点类型的列表。For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads A list of available node types can be retrieved by using the List node types API call. 此字段为必需字段。This field is required.
driver_node_type_iddriver_node_type_id STRING Spark 驱动程序的节点类型。The node type of the Spark driver. 此字段为可选;如果未设置,驱动程序节点类型将会被设置为与上面定义的 node_type_id 相同的值。This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
custom_tagscustom_tags ClusterTagClusterTag 一个对象,其中包含群集资源的一组标记。An object containing a set of tags for cluster resources. Databricks 会使用这些标记以及 default_tags 来标记所有的群集资源(如 VM)。Databricks tags all cluster resources (such as VMs) with these tags in addition to default_tags.

注意Note:

Azure Databricks 最多允许 43 个自定义标记。Azure Databricks allows at most 43 custom tags.
cluster_log_confcluster_log_conf ClusterLogConfClusterLogConf 用于将 Spark 日志传递到长期存储目标的配置。The configuration for delivering Spark logs to a long-term storage destination. 对于一个群集,只能指定一个目标。Only one destination can be specified for one cluster. 如果提供该配置,日志将会发送到目标,发送的时间间隔为If the conf is given, the logs will be delivered to the destination every
5 mins.5 mins. 驱动程序日志的目标是 <destination>/<cluster-ID>/driver,而执行程序日志的目标是 <destination>/<cluster-ID>/executorThe destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scriptsinit_scripts 一个由 InitScriptInfo 构成的数组An array of InitScriptInfo 用于存储初始化脚本的配置。The configuration for storing init scripts. 可以指定任意数量的脚本。Any number of scripts can be specified. 这些脚本会按照所提供的顺序依次执行。The scripts are executed sequentially in the order provided. 如果指定了 cluster_log_conf,初始化脚本日志将会发送到If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.<destination>/<cluster-ID>/init_scripts.
docker_imagedocker_image DockerImageDockerImage 自定义容器的 Docker 映像。Docker image for a custom container.
spark_env_varsspark_env_vars SparkEnvPairSparkEnvPair 一个对象,其中包含一组可选的由用户指定的环境变量键值对。An object containing a set of optional, user-specified environment variable key-value pairs. 在启动驱动程序和工作器时,(X,Y) 形式的键值对会按原样导出(即Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y')。export X='Y') while launching the driver and workers.

若要额外指定一组 SPARK_DAEMON_JAVA_OPTS,建议将其追加到 $SPARK_DAEMON_JAVA_OPTS,如以下示例中所示。In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. 这样可确保也包含所有默认的 databricks 托管环境变量。This ensures that all default databricks managed environmental variables are included as well.

Spark 环境变量示例:Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutesautotermination_minutes INT32 在群集处于不活动状态的时间达到此时间(以分钟为单位)后,自动终止该群集。Automatically terminates the cluster after it is inactive for this time in minutes. 如果未设置,将不会自动终止此群集。If not set, this cluster will not be automatically terminated. 如果指定,阈值必须介于 10 到 10000 分钟之间。If specified, the threshold must be between 10 and 10000 minutes. 你也可以将此值设置为 0,以显式禁用自动终止。You can also set this value to 0 to explicitly disable automatic termination.
instance_pool_idinstance_pool_id STRING 群集所属的实例池的可选 ID。The optional ID of the instance pool to which the cluster belongs. 有关详细信息,请参阅实例池 APIRefer to Instance Pools API for details.
idempotency_tokenidempotency_token STRING 可选令牌,可用于保证群集创建请求的幂等性。An optional token that can be used to guarantee the idempotency of cluster creation requests. 如果已经存在具有提供的令牌的活动群集,该请求将不会创建新群集,而是会返回现有群集的 ID。If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the ID of the existing cluster instead. 不会针对已终止的群集检查是否存在具有相同令牌的群集。The existence of a cluster with the same token is not checked against terminated clusters.

如果指定幂等性令牌,则可在失败时重试,直到该请求成功。If you specify the idempotency token, upon failure you can retry until the request succeeds. Azure Databricks 将会确保只有一个群集将通过该幂等性令牌启动。Azure Databricks will guarantee that exactly one cluster will be launched with that idempotency token.

此令牌最多只能包含 64 个字符。This token should have at most 64 characters.

响应结构 Response structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 该群集的规范标识符。Canonical identifier for the cluster.

编辑 Edit

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/edit POST

编辑群集的配置,以匹配所提供的属性和大小。Edit the configuration of a cluster to match the provided attributes and size.

如果群集处于 RUNNINGTERMINATED 状态,则可编辑该群集。You can edit a cluster if it is in a RUNNING or TERMINATED state. 如果在群集处于 RUNNING 状态时编辑该群集,它将会重启,以便使新属性能够生效。If you edit a cluster while it is in a RUNNING state, it will be restarted so that the new attributes can take effect. 如果在群集处于 TERMINATED 状态时编辑该群集,它将保持 TERMINATED 状态。If you edit a cluster while it is in a TERMINATED state, it will remain TERMINATED. 在下次使用 clusters/start API 启动它时,新属性将会生效。The next time it is started using the clusters/start API, the new attributes will take effect. 尝试编辑处于任何其他状态的群集将会被拒绝,并返回 INVALID_STATE 错误代码。An attempt to edit a cluster in any other state will be rejected with an INVALID_STATE error code.

无法编辑由 Databricks 作业服务创建的群集。Clusters created by the Databricks Jobs service cannot be edited.

示例请求:An example request:

{
 "cluster_id": "1202-211320-brick1",
 "num_workers": 10,
 "spark_version": "7.3.x-scala2.12",
 "node_type_id": "Standard_D3_v2"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

注意: 在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段将会立即更新,以反映 10 个工作器的目标大小,而 executors 中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_idcluster_id STRING 该群集的规范标识符。Canonical identifier for the cluster. 此字段为必需字段。This field is required.
cluster_namecluster_name STRING 用户请求的群集名称。Cluster name requested by the user. 此名称不必唯一。This doesn’t have to be unique. 如果在创建时未指定此字段,群集名称将为空字符串。If not specified at creation, the cluster name will be an empty string.
spark_versionspark_version STRING 群集的运行时版本。The runtime version of the cluster. 可以通过使用运行时版本 API 调用来检索可用的运行时版本的列表。You can retrieve a list of available runtime versions by using the Runtime versions API call. 此字段为必需字段。This field is required.
spark_confspark_conf SparkConfPairSparkConfPair 一个对象,其中包含一组可选的由用户指定的 Spark 配置键值对。An object containing a set of optional, user-specified Spark configuration key-value pairs. 你也可以分别通过以下属性,将额外 JVM 选项的字符串传入到驱动程序和执行程序:You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsspark.executor.extraJavaOptionsspark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

示例 Spark 配置:Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5}{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_idnode_type_id STRING 此字段通过单个值对提供给此群集中的每个 Spark 节点的资源进行编码。This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. 例如,可以针对内存密集型或计算密集型的工作负载来预配和优化 Spark 节点。通过使用列出节点类型 API 调用可以检索可用节点类型的列表。For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads A list of available node types can be retrieved by using the List node types API call. 此字段为必需字段。This field is required.
driver_node_type_iddriver_node_type_id STRING Spark 驱动程序的节点类型。The node type of the Spark driver. 此字段为可选;如果未设置,驱动程序节点类型将会被设置为与上面定义的 node_type_id 相同的值。This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
cluster_log_confcluster_log_conf ClusterLogConfClusterLogConf 用于将 Spark 日志传递到长期存储目标的配置。The configuration for delivering Spark logs to a long-term storage destination. 对于一个群集,只能指定一个目标。Only one destination can be specified for one cluster. 如果提供该配置,日志将会发送到目标,发送的时间间隔为If the conf is given, the logs will be delivered to the destination every
5 mins.5 mins. 驱动程序日志的目标是 <destination>/<cluster-ID>/driver,而执行程序日志的目标是 <destination>/<cluster-ID>/executorThe destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scriptsinit_scripts 一个由 InitScriptInfo 构成的数组An array of InitScriptInfo 用于存储初始化脚本的配置。The configuration for storing init scripts. 可以指定任意数量的目标。Any number of destinations can be specified. 这些脚本会按照所提供的顺序依次执行。The scripts are executed sequentially in the order provided. 如果指定了 cluster_log_conf,初始化脚本日志将会发送到If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.<destination>/<cluster-ID>/init_scripts.
docker_imagedocker_image DockerImageDockerImage 自定义容器的 Docker 映像。Docker image for a custom container.
spark_env_varsspark_env_vars SparkEnvPairSparkEnvPair 一个对象,其中包含一组可选的由用户指定的环境变量键值对。An object containing a set of optional, user-specified environment variable key-value pairs. 在启动驱动程序和工作器时,(X,Y) 形式的键值对会按原样导出(即Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y')。export X='Y') while launching the driver and workers.

若要额外指定一组 SPARK_DAEMON_JAVA_OPTS,建议将其追加到 $SPARK_DAEMON_JAVA_OPTS,如以下示例中所示。In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. 这可确保同时包含所有默认的 Databricks 托管环境变量。This ensures that all default Databricks managed environmental variables are included as well.

Spark 环境变量示例:Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutesautotermination_minutes INT32 在群集处于不活动状态的时间达到此时间(以分钟为单位)后,自动终止该群集。Automatically terminates the cluster after it is inactive for this time in minutes. 如果未设置,将不会自动终止此群集。If not set, this cluster will not be automatically terminated. 如果指定,阈值必须介于 10 到 10000 分钟之间。If specified, the threshold must be between 10 and 10000 minutes. 你也可以将此值设置为 0,以显式禁用自动终止。You can also set this value to 0 to explicitly disable automatic termination.
instance_pool_idinstance_pool_id STRING 群集所属的实例池的可选 ID。The optional ID of the instance pool to which the cluster belongs. 有关详细信息,请参阅Refer to Pools for details.

启动 Start

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/start POST

启动已终止的群集(如果提供了该群集的 ID)。Start a terminated cluster given its ID. 这类似于 createCluster,区别在于:This is similar to createCluster, except:

  • 已终止的群集 ID 和属性会被保留。The terminated cluster ID and attributes are preserved.
  • 该群集启动时的大小为最后指定的群集大小。The cluster starts with the last specified cluster size. 如果已终止的群集是自动缩放的群集,该群集启动时只包含最少数量的节点。If the terminated cluster is an autoscaling cluster, the cluster starts with the minimum number of nodes.
  • 如果该群集处于 RESTARTING 状态,则会返回 400 错误。If the cluster is in the RESTARTING state, a 400 error is returned.
  • 你无法启动为运行作业而启动的群集。You cannot start a cluster launched to run a job.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要启动的群集。The cluster to be started. 此字段为必需字段。This field is required.

重启 Restart

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/restart POST

如果有群集的 ID,则重启该群集。Restart a cluster given its ID. 该群集必须处于 RUNNING 状态。The cluster must be in the RUNNING state.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要启动的群集。The cluster to be started. 此字段为必需字段。This field is required.

重设大小 Resize

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/resize POST

重设群集大小,使其具有所需的工作器数。Resize a cluster to have a desired number of workers. 该群集必须处于 RUNNING 状态。The cluster must be in the RUNNING state.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1",
  "num_workers": 30
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

注意: 在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段将会立即更新,以反映 10 个工作器的目标大小,而 executors 中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_idcluster_id STRING 要重设大小的群集。The cluster to be resized. 此字段为必需字段。This field is required.

删除(终止) Delete (terminate)

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/delete POST

终止群集(如果提供了该群集的 ID)。Terminate a cluster given its ID. 该群集会以异步方式被删除。The cluster is removed asynchronously. 终止完成之后,该群集将会处于 TERMINATED 状态。Once the termination has completed, the cluster will be in the TERMINATED state. 如果该群集已处于 TERMINATINGTERMINATED 状态,则不会执行任何操作。If the cluster is already in a TERMINATING or TERMINATED state, nothing will happen.

群集在终止 30 天后会永久删除30 days after a cluster is terminated, it is permanently deleted.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要终止的群集。The cluster to be terminated. 此字段为必需字段。This field is required.

永久删除 Permanent delete

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/permanent-delete POST

永久删除群集。Permanently delete a cluster. 如果该群集正在运行,则它会被终止,并且其资源会以异步方式删除。If the cluster is running, it is terminated and its resources are asynchronously removed. 如果该群集已终止,则它会被立即删除。If the cluster is terminated, then it is immediately removed.

对于永久删除的群集,无法执行任何操作,包括检索该群集的权限。You cannot perform any action, including retrieve the cluster’s permissions, on a permanently deleted cluster. 群集列表中也不会再返回已永久删除的群集。A permanently deleted cluster is also no longer returned in the cluster list.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要永久删除的群集。The cluster to be permanently deleted. 此字段为必需字段。This field is required.

获取 Get

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/get GET

检索群集的信息(如果提供了该群集的标识符)。Retrieve the information for a cluster given its identifier. 当群集正在运行时,或在群集被终止之后的 30 天之内,可以描述群集。Clusters can be described while they are running or up to 30 days after they are terminated.

示例请求:An example request:

/clusters/get?cluster_id=1202-211320-brick1

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要检索其信息的群集。The cluster about which to retrieve information. 此字段为必需字段。This field is required.

响应结构 Response structure

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

注意: 在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段将会立即更新,以反映 10 个工作器的目标大小,而 executors 中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_idcluster_id STRING 该群集的规范标识符。Canonical identifier for the cluster. 此 ID 在群集重启和重设大小期间保留,同时每个新群集都有一个全局唯一的 ID。This ID is retained during cluster restarts and resizes, while each new cluster has a globally unique ID.
creator_user_namecreator_user_name STRING 创建者用户名。Creator user name. 如果已删除该用户,响应中将不会包括此字段。The field won’t be included in the response if the user has already been deleted.
驱动程序driver SparkNodeSparkNode Spark 驱动程序所驻留的节点。Node on which the Spark driver resides. 该驱动程序节点包含 Spark Master 和 Databricks 应用程序,该应用程序管理每个笔记本的 Spark REPL。The driver node contains the Spark master and the Databricks application that manages the per-notebook Spark REPLs.
执行程序executors 一个由 SparkNode 构成的数组An array of SparkNode Spark 执行程序所驻留的节点。Nodes on which the Spark executors reside.
spark_context_idspark_context_id INT64 规范的 SparkContext 标识符。A canonical SparkContext identifier. 此值在 Spark 驱动程序重启时会更改。This value does change when the Spark driver restarts. (cluster_id, spark_context_id) 对是所有 Spark 上下文中的全局唯一标识符。The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.
jdbc_portjdbc_port INT32 Spark JDBC 服务器用于在驱动程序节点中进行侦听的端口。Port on which Spark JDBC server is listening in the driver node. 在执行程序节点中,没有服务会在此端口上进行侦听。No service will be listening on on this port in executor nodes.
cluster_namecluster_name STRING 用户请求的群集名称。Cluster name requested by the user. 此名称不必唯一。This doesn’t have to be unique. 如果在创建时未指定此字段,群集名称将为空字符串。If not specified at creation, the cluster name will be an empty string.
spark_versionspark_version STRING 群集的运行时版本。The runtime version of the cluster. 可以通过使用运行时版本 API 调用来检索可用的运行时版本的列表。You can retrieve a list of available runtime versions by using the Runtime versions API call.
spark_confspark_conf SparkConfPairSparkConfPair 一个对象,其中包含一组可选的由用户指定的 Spark 配置键值对。An object containing a set of optional, user-specified Spark configuration key-value pairs. 你也可以分别通过以下属性,将额外 JVM 选项的字符串传入到驱动程序和执行程序:You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsspark.executor.extraJavaOptionsspark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

示例 Spark 配置:Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5}{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_idnode_type_id STRING 此字段通过单个值对提供给此群集中的每个 Spark 节点的资源进行编码。This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. 例如,可以针对内存密集型或计算密集型的工作负载来预配和优化 Spark 节点。通过使用列出节点类型 API 调用可以检索可用节点类型的列表。For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads A list of available node types can be retrieved by using the List node types API call. 此字段为必需字段。This field is required.
driver_node_type_iddriver_node_type_id STRING Spark 驱动程序的节点类型。The node type of the Spark driver. 此字段为可选;如果未设置,驱动程序节点类型将会被设置为与上面定义的 node_type_id 相同的值。This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
custom_tagscustom_tags ClusterTagClusterTag 一个对象,其中包含群集资源的一组标记。An object containing a set of tags for cluster resources. Databricks 会使用这些标记以及 default_tags 来标记所有的群集资源。Databricks tags all cluster resources with these tags in addition to default_tags.

注意Note:

* 旧版节点类型(如计算优化和内存优化)不支持标记* Tags are not supported on legacy node types such as compute-optimized and memory-optimized
* Databricks 最多允许 45 个自定义标记* Databricks allows at most 45 custom tags
cluster_log_confcluster_log_conf ClusterLogConfClusterLogConf 用于将 Spark 日志传递到长期存储目标的配置。The configuration for delivering Spark logs to a long-term storage destination. 对于一个群集,只能指定一个目标。Only one destination can be specified for one cluster. 如果提供该配置,日志将会发送到目标,发送的时间间隔为If the conf is given, the logs will be delivered to the destination every
5 mins.5 mins. 驱动程序日志的目标是 <destination>/<cluster-ID>/driver,而执行程序日志的目标是 <destination>/<cluster-ID>/executorThe destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scriptsinit_scripts 一个由 InitScriptInfo 构成的数组An array of InitScriptInfo 用于存储初始化脚本的配置。The configuration for storing init scripts. 可以指定任意数量的目标。Any number of destinations can be specified. 这些脚本会按照所提供的顺序依次执行。The scripts are executed sequentially in the order provided. 如果指定了 cluster_log_conf,初始化脚本日志将会发送到If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.<destination>/<cluster-ID>/init_scripts.
docker_imagedocker_image DockerImageDockerImage 自定义容器的 Docker 映像。Docker image for a custom container.
spark_env_varsspark_env_vars SparkEnvPairSparkEnvPair 一个对象,其中包含一组可选的由用户指定的环境变量键值对。An object containing a set of optional, user-specified environment variable key-value pairs. 在启动驱动程序和工作器时,(X,Y) 形式的键值对会按原样导出(即Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y')。export X='Y') while launching the driver and workers.

若要额外指定一组 SPARK_DAEMON_JAVA_OPTS,建议将其追加到 $SPARK_DAEMON_JAVA_OPTS,如以下示例中所示。In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. 这可确保同时包含所有默认的 Databricks 托管环境变量。This ensures that all default Databricks managed environmental variables are included as well.

Spark 环境变量示例:Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutesautotermination_minutes INT32 在群集处于不活动状态的时间达到此时间(以分钟为单位)后,自动终止该群集。Automatically terminates the cluster after it is inactive for this time in minutes. 如果未设置,将不会自动终止此群集。If not set, this cluster will not be automatically terminated. 如果指定,阈值必须介于 10 到 10000 分钟之间。If specified, the threshold must be between 10 and 10000 minutes. 你也可以将此值设置为 0,以显式禁用自动终止。You can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_diskenable_elastic_disk BOOL 自动缩放本地存储:启用后,此群集在其 Spark 工作器磁盘空间不足时将会动态获取更多磁盘空间。Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. 有关详细信息,请参阅自动缩放本地存储See Autoscaling local storage for details.
instance_pool_idinstance_pool_id STRING 群集所属的实例池的可选 ID。The optional ID of the instance pool to which the cluster belongs. 有关详细信息,请参阅Refer to Pools for details.
statestate ClusterStateClusterState 群集的状态。State of the cluster.
state_messagestate_message STRING 与最新状态转换关联的消息(例如,群集进入 TERMINATED 状态的原因)。A message associated with the most recent state transition (for example, the reason why the cluster entered the TERMINATED state).
start_timestart_time INT64 (在群集进入 PENDING 状态时)收到群集创建请求的时间(以 epoch 毫秒表示)。Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered the PENDING state).
terminated_timeterminated_time INT64 终止群集(如果适用)的时间,以 epoch 毫秒表示。Time (in epoch milliseconds) when the cluster was terminated, if applicable.
last_state_loss_timelast_state_loss_time INT64 群集驱动程序上一次丢失其状态(由于重启或驱动程序故障)的时间。Time when the cluster driver last lost its state (due to a restart or driver failure).
last_activity_timelast_activity_time INT64 群集上次处于活动状态的时间(以 epoch 毫秒表示)。Time (in epoch milliseconds) when the cluster was last active. 如果群集上至少有一个命令尚未完成,则该群集处于活动状态。A cluster is active if there is at least one command that has not finished on the cluster. 此字段在群集已到达 RUNNING 状态之后可用。This field is available after the cluster has reached the RUNNING state. 我们会尽最大努力尝试对此字段进行更新。Updates to this field are made as best-effort attempts. 某些版本的 Spark 不支持报告群集活动性。Certain versions of Spark do not support reporting of cluster activity. 有关详细信息,请参阅自动终止Refer to Automatic termination for details.
cluster_memory_mbcluster_memory_mb INT64 以兆字节表示的群集内存总量。Total amount of cluster memory, in megabytes.
cluster_corescluster_cores FLOAT 可用于此群集的 CPU 内核数。Number of CPU cores available for this cluster. 这可以是小数,因为某些节点类型被配置为在同一实例上的 Spark 节点之间共享内核。This can be fractional since certain node types are configured to share cores between Spark nodes on the same instance.
default_tagsdefault_tags ClusterTagClusterTag 一个对象,其中包含一组由 Azure Databricks 添加(不管任何 custom_tags 如何)的标记,包括:An object containing a set of tags that are added by Azure Databricks regardless of any custom_tags, including:

* Vendor:Databricks* Vendor: Databricks
* Creator: * Creator:
* ClusterName: * ClusterName:
* ClusterId: * ClusterId:
* Name: 在作业群集上:* Name: On job clusters:

* RunName: * RunName:
* JobId: * JobId:
cluster_log_statuscluster_log_status LogSyncStatusLogSyncStatus 群集日志发送状态。Cluster log delivery status.
termination_reasontermination_reason TerminationReasonTerminationReason 有关群集终止原因的信息。Information about why the cluster was terminated. 此字段只在群集处于 TERMINATINGTERMINATED 状态时才会显示。This field appears only when the cluster is in the TERMINATING or TERMINATED state.

固定 Pin

备注

只有 Azure Databricks 管理员才能调用此 API。You must be an Azure Databricks administrator to invoke this API.

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/pin POST

确保即使在群集已被终止的时间超过了 30 天的情况下仍会保留通用群集配置。Ensure that an all-purpose cluster configuration is retained even after a cluster has been terminated for more than 30 days. “固定”可确保群集始终会由列表 API 返回。Pinning ensures that the cluster is always returned by the List API. 固定一个已经固定的群集不会起作用。Pinning a cluster that is already pinned has no effect.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要固定的群集。The cluster to pin. 此字段为必需字段。This field is required.

取消固定 Unpin

备注

只有 Azure Databricks 管理员才能调用此 API。You must be an Azure Databricks administrator to invoke this API.

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/unpin POST

允许最终从列表 API 返回的列表中删除群集。Allows the cluster to eventually be removed from the list returned by the List API. 对一个未固定的群集取消固定不会起作用。Unpinning a cluster that is not pinned has no effect.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

请求结构 Request structure

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要取消固定的群集。The cluster to unpin. 此字段为必需字段。This field is required.

列表 List

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/list GET

返回有关所有固定群集、活动群集、过去 30 天内最多 70 个最近终止的通用群集以及过去 30 天内最多 30 个最近终止的作业群集的相关信息。Return information about all pinned clusters, active clusters, up to 70 of the most recently terminated all-purpose clusters in the past 30 days, and up to 30 of the most recently terminated job clusters in the past 30 days. 例如,如果有 1 个固定群集、4 个活动群集、45 个在过去 30 天内终止的通用群集以及 50 个在过去 30 天内终止的作业群集,则此 API 会返回 1 个固定群集、4 个活动群集、所有 45 个终止的通用群集,以及 30 个最近终止的作业群集。For example, if there is 1 pinned cluster, 4 active clusters, 45 terminated all-purpose clusters in the past 30 days, and 50 terminated job clusters in the past 30 days, then this API returns the 1 pinned cluster, 4 active clusters, all 45 terminated all-purpose clusters, and the 30 most recently terminated job clusters.

响应结构 Response structure

字段名称Field Name 类型Type 描述Description
clustersclusters 一个由 ClusterInfo 构成的数组An array of ClusterInfo 群集的列表。A list of clusters.

列出节点类型 List node types

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/list-node-types GET

返回受支持的 Spark 节点类型的列表。Return a list of supported Spark node types. 这些节点类型可用于启动群集。These node types can be used to launch a cluster.

响应结构 Response structure

字段名称Field Name 类型Type 描述Description
node_typesnode_types 一个由 NodeType 构成的数组An array of NodeType 可用的 Spark 节点类型的列表。The list of available Spark node types.

运行时版本 Runtime versions

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/spark-versions GET

返回可用的运行时版本的列表。Return the list of available runtime versions. 这些版本可用于启动群集。These versions can be used to launch a cluster.

响应结构 Response structure

字段名称Field Name 类型Type 描述Description
versionsversions 一个由 SparkVersion 构成的数组An array of SparkVersion 所有可用的运行时版本。All the available runtime versions.

事件 Events

端点Endpoint HTTP 方法HTTP Method
2.0/clusters/events POST

检索有关群集活动性的事件列表。Retrieve a list of events about the activity of a cluster. 你可以检索活动的(正在运行的、挂起的或正在重新配置的)群集和已终止的群集(自其上次终止后 30 天内)中的事件。You can retrieve events from active clusters (running, pending, or reconfiguring) and terminated clusters within 30 days of their last termination. 此 API 支持分页。This API is paginated. 如果有更多事件需要读取,响应会包括用于请求下一页事件所需的所有参数。If there are more events to read, the response includes all the parameters necessary to request the next page of events.

示例请求:An example request:

{
  "cluster_id": "1202-211320-brick1"
}

示例响应:An example response:

{
  "events": [{
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1534371918659,
    "type": "TERMINATING",
    "details": {
      "reason": {
        "code": "INACTIVITY",
        "parameters": {
          "inactivity_duration_min": "120"
        }
      }
    }
  }, {
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1534358289590,
    "type": "RUNNING",
    "details": {
      "current_num_workers": 2,
      "target_num_workers": 2
    }
  }, {
    "cluster_id": "1202-211320-brick1",
    "timestamp": 1533225298406,
    "type": "RESTARTING",
    "details": {
      "user": "admin"
    }
  }],
  "next_page": {
    "cluster_id": "0802-034608-aloe926",
    "end_time": 1534371918659,
    "offset": 50
  },
  "total_count": 55
}

用于检索下一页事件的请求示例:Example request to retrieve the next page of events:

{
  "cluster_id": "1202-211320",
  "start_time": 1534371918659
}

请求结构 Request structure

检索与某个特定群集相关的事件。Retrieve events pertaining to a specific cluster.

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 要检索其相关事件的群集的 ID。The ID of the cluster to retrieve events about. 此字段为必需字段。This field is required.
start_timestart_time INT64 以 epoch 毫秒表示的开始时间。The start time in epoch milliseconds. 如果为空,则返回从起始时间开始的事件。If empty, returns events starting from the beginning of time.
end_timeend_time INT64 以 epoch 毫秒表示的结束时间。The end time in epoch milliseconds. 如果为空,则返回截止到当前时间的事件。If empty, returns events up to the current time.
顺序order ListOrderListOrder 列出事件时采用的顺序;ASCDESCThe order to list events in; either ASC or DESC. 默认为 DESCDefaults to DESC.
event_typesevent_types 一个由 ClusterEventType 构成的数组An array of ClusterEventType 可选的一组要筛选的事件类型。An optional set of event types to filter on. 如果为空,则返回所有事件类型。If empty, all event types are returned.
offsetoffset INT64 结果集之中的偏移量。The offset in the result set. 默认为 0(无偏移)。Defaults to 0 (no offset). 如果指定了偏移量,并按降序请求结果,则“end_time”字段为必填字段。When an offset is specified and the results are requested in descending order, the end_time field is required.
limitlimit INT64 要包括在一页事件中的最大事件数。The maximum number of events to include in a page of events. 默认为 50,允许的最大值为 500。Defaults to 50, and maximum allowed value is 500.

响应结构 Response structure

字段名称Field Name 类型Type 说明Description
活动events 一个由 ClusterEvent 构成的数组An array of ClusterEvent 匹配事件的列表。This list of matching events.
next_pagenext_page 请求结构Request structure 检索下一页事件所需的参数。The parameters required to retrieve the next page of events. 如果没有更多要读取的事件,则省略此字段。Omitted if there are no more events to read.
total_counttotal_count INT64 按 Start_time、end_time 和 event_types 筛选的事件的总数。The total number of events filtered by the start_time, end_time, and event_types.

数据结构 Data structures

本节内容:In this section:

AutoScale AutoScale

定义群集工作器的最小数量和最大数量的范围。Range defining the min and max number of cluster workers.

字段名称Field Name 类型Type 描述Description
min_workersmin_workers INT32 群集在未充分利用时可纵向缩减到的最小工作器数。The minimum number of workers to which the cluster can scale down when underutilized. 此数量也是群集在创建后将会具有的初始工作器的数量。It is also the initial number of workers the cluster will have after creation.
max_workersmax_workers INT32 群集在负载过高时可纵向扩展到的最大工作器数。The maximum number of workers to which the cluster can scale up when overloaded. max_workers 必须严格大于 min_workers。max_workers must be strictly greater than min_workers.

ClusterInfo ClusterInfo

有关群集的元数据。Metadata about a cluster.

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

注意: 在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段将会立即更新,以反映 10 个工作器的目标大小,而 executors 中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_idcluster_id STRING 该群集的规范标识符。Canonical identifier for the cluster. 此 ID 在群集重启和重设大小期间保留,同时每个新群集都有一个全局唯一的 ID。This ID is retained during cluster restarts and resizes, while each new cluster has a globally unique ID.
creator_user_namecreator_user_name STRING 创建者用户名。Creator user name. 如果已删除该用户,响应中将不会包括此字段。The field won’t be included in the response if the user has already been deleted.
驱动程序driver SparkNodeSparkNode Spark 驱动程序所驻留的节点。Node on which the Spark driver resides. 该驱动程序节点包含 Spark Master 和 Databricks 应用程序,该应用程序管理每个笔记本的 Spark REPL。The driver node contains the Spark master and the Databricks application that manages the per-notebook Spark REPLs.
执行程序executors 一个由 SparkNode 构成的数组An array of SparkNode Spark 执行程序所驻留的节点。Nodes on which the Spark executors reside.
spark_context_idspark_context_id INT64 规范的 SparkContext 标识符。A canonical SparkContext identifier. 此值在 Spark 驱动程序重启时会更改。This value does change when the Spark driver restarts. (cluster_id, spark_context_id) 对是所有 Spark 上下文中的全局唯一标识符。The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.
jdbc_portjdbc_port INT32 Spark JDBC 服务器用于在驱动程序节点中进行侦听的端口。Port on which Spark JDBC server is listening in the driver node. 在执行程序节点中,没有服务会在此端口上进行侦听。No service will be listening on on this port in executor nodes.
cluster_namecluster_name STRING 用户请求的群集名称。Cluster name requested by the user. 此名称不必唯一。This doesn’t have to be unique. 如果在创建时未指定此字段,群集名称将为空字符串。If not specified at creation, the cluster name will be an empty string.
spark_versionspark_version STRING 群集的运行时版本。The runtime version of the cluster. 可以通过使用运行时版本 API 调用来检索可用的运行时版本的列表。You can retrieve a list of available runtime versions by using the Runtime versions API call.
spark_confspark_conf SparkConfPairSparkConfPair 一个对象,其中包含一组可选的由用户指定的 Spark 配置键值对。An object containing a set of optional, user-specified Spark configuration key-value pairs. 你也可以分别通过以下属性,将额外 JVM 选项的字符串传入到驱动程序和执行程序:You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsspark.executor.extraJavaOptionsspark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

示例 Spark 配置:Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5}{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_idnode_type_id STRING 此字段通过单个值对提供给此群集中的每个 Spark 节点的资源进行编码。This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. 例如,可以针对内存或计算密集型工作负载对 Spark 节点进行预配和优化。For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. 通过使用列出节点类型 API 调用,可以检索可用节点类型的列表。A list of available node types can be retrieved by using the List node types API call.
driver_node_type_iddriver_node_type_id STRING Spark 驱动程序的节点类型。The node type of the Spark driver. 此字段为可选;如果未设置,驱动程序节点类型将会被设置为与上面定义的 node_type_id 相同的值。This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
cluster_log_confcluster_log_conf ClusterLogConfClusterLogConf 用于将 Spark 日志传递到长期存储目标的配置。The configuration for delivering Spark logs to a long-term storage destination. 对于一个群集,只能指定一个目标。Only one destination can be specified for one cluster. 如果提供该配置,日志将会发送到目标,发送的时间间隔为If the conf is given, the logs will be delivered to the destination every
5 mins.5 mins. 驱动程序日志的目标是 <destination>/<cluster-ID>/driver,而执行程序日志的目标是 <destination>/<cluster-ID>/executorThe destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scriptsinit_scripts 一个由 InitScriptInfo 构成的数组An array of InitScriptInfo 用于存储初始化脚本的配置。The configuration for storing init scripts. 可以指定任意数量的目标。Any number of destinations can be specified. 这些脚本会按照所提供的顺序依次执行。The scripts are executed sequentially in the order provided. 如果指定了 cluster_log_conf,初始化脚本日志将会发送到If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.<destination>/<cluster-ID>/init_scripts.
docker_imagedocker_image DockerImageDockerImage 自定义容器的 Docker 映像。Docker image for a custom container.
spark_env_varsspark_env_vars SparkEnvPairSparkEnvPair 一个对象,其中包含一组可选的由用户指定的环境变量键值对。An object containing a set of optional, user-specified environment variable key-value pairs. 在启动驱动程序和工作器时,(X,Y) 形式的键值对会按原样导出(即Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y')。export X='Y') while launching the driver and workers.

若要额外指定一组 SPARK_DAEMON_JAVA_OPTS,建议将其追加到 $SPARK_DAEMON_JAVA_OPTS,如以下示例中所示。To specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. 这样就确保了还会包含所有默认的 databricks 托管环境变量。This ensures that all default databricks managed environmental variables are included as well.

Spark 环境变量示例:Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutesautotermination_minutes INT32 在群集处于不活动状态的时间达到此时间(以分钟为单位)后,自动终止该群集。Automatically terminates the cluster after it is inactive for this time in minutes. 如果未设置,将不会自动终止此群集。If not set, this cluster will not be automatically terminated. 如果指定,阈值必须介于 10 到 10000 分钟之间。If specified, the threshold must be between 10 and 10000 minutes. 你也可以将此值设置为 0,以显式禁用自动终止。You can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_diskenable_elastic_disk BOOL 自动缩放本地存储:启用后,此群集在其 Spark 工作器磁盘空间不足时将会动态获取更多磁盘空间。Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. 有关详细信息,请参阅自动缩放本地存储See Autoscaling local storage for details.
instance_pool_idinstance_pool_id STRING 群集所属的实例池的可选 ID。The optional ID of the instance pool to which the cluster belongs. 有关详细信息,请参阅Refer to Pools for details.
statestate ClusterStateClusterState 群集的状态。State of the cluster.
state_messagestate_message STRING 与最近的状态转换关联的消息(例如,群集进入 TERMINATED 状态的原因)。A message associated with the most recent state transition (for example, the reason why the cluster entered a TERMINATED state).
start_timestart_time INT64 (在群集进入 PENDING 状态时)收到群集创建请求的时间(以 epoch 毫秒表示)。Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered a PENDING state).
terminated_timeterminated_time INT64 终止群集(如果适用)的时间,以 epoch 毫秒表示。Time (in epoch milliseconds) when the cluster was terminated, if applicable.
last_state_loss_timelast_state_loss_time INT64 群集驱动程序上一次丢失其状态(由于重启或驱动程序故障)的时间。Time when the cluster driver last lost its state (due to a restart or driver failure).
last_activity_timelast_activity_time INT64 群集上次处于活动状态的时间(以 epoch 毫秒表示)。Time (in epoch milliseconds) when the cluster was last active. 如果群集上至少有一个命令尚未完成,则该群集处于活动状态。A cluster is active if there is at least one command that has not finished on the cluster. 此字段在群集已到达 RUNNING 状态之后可用。This field is available after the cluster has reached a RUNNING state. 我们会尽最大努力尝试对此字段进行更新。Updates to this field are made as best-effort attempts. 某些版本的 Spark 不支持报告群集活动性。Certain versions of Spark do not support reporting of cluster activity. 有关详细信息,请参阅自动终止Refer to Automatic termination for details.
cluster_memory_mbcluster_memory_mb INT64 以兆字节表示的群集内存总量。Total amount of cluster memory, in megabytes.
cluster_corescluster_cores FLOAT 可用于此群集的 CPU 内核数。Number of CPU cores available for this cluster. 这可以是小数,因为某些节点类型被配置为在同一实例上的 Spark 节点之间共享内核。This can be fractional since certain node types are configured to share cores between Spark nodes on the same instance.
default_tagsdefault_tags ClusterTagClusterTag 一个对象,其中包含一组由 Azure Databricks 添加(不管任何 custom_tags 如何)的标记,包括:An object containing a set of tags that are added by Azure Databricks regardless of any custom_tags, including:

* Vendor:Databricks* Vendor: Databricks
* Creator: * Creator:
* ClusterName: * ClusterName:
* ClusterId: * ClusterId:
* Name: 在作业群集上:* Name: On job clusters:

* RunName: * RunName:
* JobId: * JobId:
cluster_log_statuscluster_log_status LogSyncStatusLogSyncStatus 群集日志发送状态。Cluster log delivery status.
termination_reasontermination_reason TerminationReasonTerminationReason 有关群集终止原因的信息。Information about why the cluster was terminated. 此字段只在群集处于 TERMINATINGTERMINATED 状态时才会出现。This field only appears when the cluster is in a TERMINATING or TERMINATED state.

ClusterEvent ClusterEvent

群集事件信息。Cluster event information.

字段名称Field Name 类型Type 描述Description
cluster_idcluster_id STRING 该群集的规范标识符。Canonical identifier for the cluster. 此字段为必需字段。This field is required.
timestamptimestamp INT64 事件发生时的时间戳,存储为自 unix epoch 以来的毫秒数。The timestamp when the event occurred, stored as the number of milliseconds since the unix epoch. 由 Timeline 服务分配。Assigned by the Timeline service.
typetype ClusterEventTypeClusterEventType 事件类型。The event type. 此字段为必需字段。This field is required.
详细信息details EventDetailsEventDetails 事件详细信息。The event details. 此字段为必需字段。This field is required.

ClusterEventType ClusterEventType

群集事件的类型。Type of a cluster event.

事件类型Event Type 说明Description
CREATINGCREATING 指示正在创建群集。Indicates that the cluster is being created.
DID_NOT_EXPAND_DISKDID_NOT_EXPAND_DISK 指示磁盘空间不足,但添加磁盘会超出最大容量。Indicates that a disk is low on space, but adding disks would put it over the max capacity.
EXPANDED_DISKEXPANDED_DISK 指示磁盘空间不足,并且磁盘已扩展。Indicates that a disk was low on space and the disks were expanded.
FAILED_TO_EXPAND_DISKFAILED_TO_EXPAND_DISK 指示磁盘空间不足,并且无法扩展磁盘。Indicates that a disk was low on space and disk space could not be expanded.
INIT_SCRIPTS_STARTINGINIT_SCRIPTS_STARTING 指示群集范围的初始化脚本已启动。Indicates that the cluster scoped init script has started.
INIT_SCRIPTS_FINISHEDINIT_SCRIPTS_FINISHED 指示群集范围的初始化脚本已完成。Indicates that the cluster scoped init script has finished.
STARTINGSTARTING 指示正在启动群集。Indicates that the cluster is being started.
RESTARTINGRESTARTING 指示正在重启群集。Indicates that the cluster is being started.
TERMINATINGTERMINATING 指示正在终止群集。Indicates that the cluster is being terminated.
EDITEDEDITED 指示已编辑了群集。Indicates that the cluster has been edited.
RUNNINGRUNNING 指示已完成群集创建。Indicates the cluster has finished being created. 包括群集中的节点数,以及失败原因(如果无法获取某些节点)。Includes the number of nodes in the cluster and a failure reason if some nodes could not be acquired.
RESIZINGRESIZING 指示群集的目标大小的变化(变大或变小)。Indicates a change in the target size of the cluster (upsize or downsize).
UPSIZE_COMPLETEDUPSIZE_COMPLETED 指示已完成将节点添加到群集。Indicates that nodes finished being added to the cluster. 包括群集中的节点数,以及失败原因(如果无法获取某些节点)。Includes the number of nodes in the cluster and a failure reason if some nodes could not be acquired.
NODES_LOSTNODES_LOST 指示某些节点已从群集中丢失。Indicates that some nodes were lost from the cluster.
DRIVER_HEALTHYDRIVER_HEALTHY 指示驱动程序运行正常且群集已可供使用。Indicates that the driver is healthy and the cluster is ready for use.
DRIVER_UNAVAILABLEDRIVER_UNAVAILABLE 指示该驱动程序不可用。Indicates that the driver is unavailable.
SPARK_EXCEPTIONSPARK_EXCEPTION 指示从驱动程序中引发了 Spark 异常。Indicates that a Spark exception was thrown from the driver.
DRIVER_NOT_RESPONDINGDRIVER_NOT_RESPONDING 指示驱动程序已启动,但没有响应(可能是由于 GC)。Indicates that the driver is up but is not responsive, likely due to GC.
DBFS_DOWNDBFS_DOWN 指示驱动程序已启动,但 DBFS 已关闭。Indicates that the driver is up but DBFS is down.
METASTORE_DOWNMETASTORE_DOWN 指示驱动程序已启动,但元存储已关闭。Indicates that the driver is up but the metastore is down.
NODE_BLACKLISTEDNODE_BLACKLISTED 指示 Spark 不允许使用某个节点。Indicates that a node is not allowed by Spark.
PINNEDPINNED 指示群集已固定。Indicates that the cluster was pinned.
UNPINNEDUNPINNED 指示群集已取消固定。Indicates that the cluster was unpinned.

EventDetails EventDetails

有关群集事件的详细信息。Details about a cluster event.

字段名称Field Name 类型Type 描述Description
current_num_workerscurrent_num_workers INT32 群集中节点的数量。The number of nodes in the cluster.
target_num_workerstarget_num_workers INT32 群集中节点的目标数量。The targeted number of nodes in the cluster.
previous_attributesprevious_attributes ClusterAttributesClusterAttributes 编辑群集之前的群集属性。The cluster attributes before a cluster was edited.
attributesattributes ClusterAttributesClusterAttributes * 对于已创建的群集,此项为群集的属性。* For created clusters, the attributes of the cluster.
* 对于已编辑的群集,此项为群集的新属性。* For edited clusters, the new attributes of the cluster.
previous_cluster_sizeprevious_cluster_size clusterSizeClusterSize 群集在编辑或重设大小之前的大小。The size of the cluster before an edit or resize.
cluster_sizecluster_size clusterSizeClusterSize 在创建或编辑群集时所设置的群集大小。The cluster size that was set in the cluster creation or edit.
causecause ResizeCauseResizeCause 目标大小发生变化的原因。The cause of a change in target size.
reasonreason TerminationReasonTerminationReason 终止原因:A termination reason:

* 对于 TERMINATED 事件,此项为终止的原因。* On a TERMINATED event, the reason for the termination.
* 对于 RESIZE_COMPLETE 事件,此项指示未能获取某些节点的原因。* On a RESIZE_COMPLETE event, indicates the reason that we failed to acquire some nodes.
useruser STRING 导致事件发生的用户。The user that caused the event to occur. (如果事件由 Azure Databricks 导致,则此项为空。)(Empty if it was done by Azure Databricks.)

ClusterAttributes ClusterAttributes

在群集创建过程中设置的一组常见属性。Common set of attributes set during cluster creation. 在群集的生存期内无法更改这些属性。These attributes cannot be changed over the lifetime of a cluster.

字段名称Field Name 类型Type 描述Description
cluster_namecluster_name STRING 用户请求的群集名称。Cluster name requested by the user. 此名称不必唯一。This doesn’t have to be unique. 如果在创建时未指定此字段,群集名称将为空字符串。If not specified at creation, the cluster name will be an empty string.
spark_versionspark_version STRING 群集的运行时版本,例如“5.0.x-scala2.11”。The runtime version of the cluster, for example “5.0.x-scala2.11”. 可以通过使用运行时版本 API 调用来检索可用的运行时版本的列表。You can retrieve a list of available runtime versions by using the Runtime versions API call.
spark_confspark_conf SparkConfPairSparkConfPair 一个对象,其中包含一组可选的由用户指定的 Spark 配置键值对。An object containing a set of optional, user-specified Spark configuration key-value pairs. 你也可以分别通过以下属性,将额外 JVM 选项的字符串传入到驱动程序和执行程序:You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsspark.executor.extraJavaOptionsspark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

示例 Spark 配置:Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5}{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_idnode_type_id STRING 此字段通过单个值对提供给此群集中的每个 Spark 节点的资源进行编码。This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. 例如,可以针对内存密集型或计算密集型的工作负载来预配和优化 Spark 节点。通过使用列出节点类型 API 调用可以检索可用节点类型的列表。For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads A list of available node types can be retrieved by using the List node types API call.
driver_node_type_iddriver_node_type_id STRING Spark 驱动程序的节点类型。The node type of the Spark driver. 此字段为可选;如果未设置,驱动程序节点类型将会被设置为与上面定义的 node_type_id 相同的值。This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
ssh_public_keysssh_public_keys 一个由 STRING 构成的数组An array of STRING 将会添加到此群集中各个 Spark 节点的 SSH 公钥内容。SSH public key contents that will be added to each Spark node in this cluster. 对应的私钥可用于在端口 2200上使用用户名 ubuntu 登录。The corresponding private keys can be used to login with the user name ubuntu on port 2200. 最多可以指定 10 个密钥。Up to 10 keys can be specified.
custom_tagscustom_tags ClusterTagClusterTag 一个对象,其中包含群集资源的一组标记。An object containing a set of tags for cluster resources. Databricks 会使用这些标记以及 default_tags 来标记所有的群集资源。Databricks tags all cluster resources with these tags in addition to default_tags.

注意Note:

* 旧版节点类型(如计算优化和内存优化)不支持标记* Tags are not supported on legacy node types such as compute-optimized and memory-optimized
* Databricks 最多允许 45 个自定义标记* Databricks allows at most 45 custom tags
cluster_log_confcluster_log_conf ClusterLogConfClusterLogConf 用于将 Spark 日志传递到长期存储目标的配置。The configuration for delivering Spark logs to a long-term storage destination. 对于一个群集,只能指定一个目标。Only one destination can be specified for one cluster. 如果提供该配置,日志将会发送到目标,发送的时间间隔为If the conf is given, the logs will be delivered to the destination every
5 mins.5 mins. 驱动程序日志的目标是 <destination>/<cluster-ID>/driver,而执行程序日志的目标是 <destination>/<cluster-ID>/executorThe destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scriptsinit_scripts 一个由 InitScriptInfo 构成的数组An array of InitScriptInfo 用于存储初始化脚本的配置。The configuration for storing init scripts. 可以指定任意数量的目标。Any number of destinations can be specified. 这些脚本会按照所提供的顺序依次执行。The scripts are executed sequentially in the order provided. 如果指定了 cluster_log_conf,初始化脚本日志将会发送到If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.<destination>/<cluster-ID>/init_scripts.
docker_imagedocker_image DockerImageDockerImage 自定义容器的 Docker 映像。Docker image for a custom container.
spark_env_varsspark_env_vars SparkEnvPairSparkEnvPair 一个对象,其中包含一组可选的由用户指定的环境变量键值对。An object containing a set of optional, user-specified environment variable key-value pairs. 在启动驱动程序和工作器时,(X,Y) 形式的键值对会按原样导出(即Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y')。export X='Y') while launching the driver and workers.

若要额外指定一组 SPARK_DAEMON_JAVA_OPTS,建议将其追加到 $SPARK_DAEMON_JAVA_OPTS,如以下示例中所示。In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. 这样可确保也包含所有默认的 databricks 托管环境变量。This ensures that all default databricks managed environmental variables are included as well.

Spark 环境变量示例:Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutesautotermination_minutes INT32 在群集处于不活动状态的时间达到此时间(以分钟为单位)后,自动终止该群集。Automatically terminates the cluster after it is inactive for this time in minutes. 如果未设置,将不会自动终止此群集。If not set, this cluster will not be automatically terminated. 如果指定,阈值必须介于 10 到 10000 分钟之间。If specified, the threshold must be between 10 and 10000 minutes. 你也可以将此值设置为 0,以显式禁用自动终止。You can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_diskenable_elastic_disk BOOL 自动缩放本地存储:启用后,此群集在其 Spark 工作器磁盘空间不足时将会动态获取更多磁盘空间。Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. 有关详细信息,请参阅自动缩放本地存储See Autoscaling local storage for details.
instance_pool_idinstance_pool_id STRING 群集所属的实例池的可选 ID。The optional ID of the instance pool to which the cluster belongs. 有关详细信息,请参阅Refer to Pools for details.
cluster_sourcecluster_source ClusterSourceClusterSource 确定群集是由用户通过 UI 创建的,还是由 Databricks 作业计划程序创建的,还是通过 API 请求创建的。Determines whether the cluster was created by a user through the UI, created by the Databricks Jobs scheduler, or through an API request.
policy_idpolicy_id STRING 群集策略 ID。A cluster policy ID.

ClusterSize ClusterSize

群集大小规范。Cluster size specification.

字段名称Field Name 类型Type 描述Description
num_workers 或 autoscalenum_workers OR autoscale INT32AutoScaleINT32 OR AutoScale 如果是 num_workers,则此项为此群集应该具有的工作器节点数。If num_workers, number of worker nodes that this cluster should have. 一个群集有一个 Spark 驱动程序和 num_workers 个执行程序用于总共 (num_workers + 1) 个 Spark 节点。A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

在读取群集的属性时,此字段反映的是所需的工作器数,而不是实际的工作器数。When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. 例如,如果将群集的大小从 5 个工作器重设为 10 个工作器,此字段会更新,以反映 10 个工作器的目标大小,而执行程序中列出的工作器将会随着新节点的预配,逐渐从 5 个增加到 10 个。For instance, if a cluster is resized from 5 to 10 workers, this field is updated to reflect the target size of 10 workers, whereas the workers listed in executors gradually increase from 5 to 10 as the new nodes are provisioned.

如果是 autoscale,则会需要参数,以便根据负载自动纵向扩展或缩减群集。If autoscale, parameters needed in order to automatically scale clusters up and down based on load.

ListOrder ListOrder

基于列表的查询的泛型排序枚举。Generic ordering enum for list-based queries.

订单Order 描述Description
DESCDESC 降序排序。Descending order.
ASCASC 升序排序。Ascending order.

ResizeCause ResizeCause

重设群集大小的原因。Reason why a cluster was resized.

原因Cause 说明Description
AUTOSCALEAUTOSCALE 根据负载自动重设了大小。Automatically resized based on load.
USER_REQUESTUSER_REQUEST 用户请求了新的大小。User requested a new size.
AUTORECOVERYAUTORECOVERY 在群集丢失节点后,自动恢复监视器重设了该群集的大小。Autorecovery monitor resized the cluster after it lost a node.

ClusterLogConf ClusterLogConf

群集日志的路径。Path to cluster log.

字段名称Field Name 类型Type 描述Description
dbfsdbfs DbfsStorageInfoDbfsStorageInfo 群集日志的 DBFS 位置。DBFS location of cluster log. 必须提供目标。Destination must be provided. 例如,应用于对象的For example,
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

InitScriptInfo InitScriptInfo

初始化脚本的路径。Path to an init script. 若要了解如何将初始化脚本与 Databricks 容器服务配合使用,请参阅使用初始化脚本For instructions on using init scripts with Databricks Container Services, see Use an init script.

备注

文件存储类型只适用于使用 Databricks 容器服务设置的群集。The file storage type is only available for clusters set up using Databricks Container Services.

字段名称Field Name 类型Type 描述Description
dbfs 或 filedbfs OR file DbfsStorageInfoDbfsStorageInfo

FileStorageInfoFileStorageInfo
初始化脚本的 DBFS 位置。DBFS location of init script. 必须提供目标。Destination must be provided. 例如,应用于对象的For example,
{ "dbfs" : { "destination" : "dbfs:/home/init_script" } }

初始化脚本的文件位置。File location of init script. 必须提供目标。Destination must be provided. 例如,应用于对象的For example,
{ "file" : { "destination" : "file:/my/local/file.sh" } }

ClusterTag ClusterTag

群集标记定义。Cluster tag definition.

类型Type 描述Description
STRING 标记的键。The key of the tag. 该键必须符合以下要求:The key must:

* 长度介于 1 到 512 个字符之间* Be between 1 and 512 characters long
* 未包含 <>%*&+?\\/ 中的任何字符* Not contain any of the characters <>%*&+?\\/
* 不以 azuremicrosoftwindows 开头* Not begin with azure, microsoft, or windows
STRING 标记的值。The value of the tag. 值的长度必须小于或等于 256 个 UTF-8 字符。The value length must be less than or equal to 256 UTF-8 characters.

DbfsStorageInfo DbfsStorageInfo

DBFS 存储信息。DBFS storage information.

字段名称Field Name 类型Type 描述Description
目标destination STRING DBFS 目标。DBFS destination. 示例: dbfs:/my/pathExample: dbfs:/my/path

FileStorageInfo FileStorageInfo

文件存储信息。File storage information.

备注

此位置类型只适用于使用 Databricks 容器服务设置的群集。This location type is only available for clusters set up using Databricks Container Services.

字段名称Field Name 类型Type 描述Description
目标destination STRING 文件目标。File destination. 示例: file:/my/file.shExample: file:/my/file.sh

DockerImageDockerImage

Docker 映像连接信息。Docker image connection information.

字段Field 类型Type 描述Description
urlurl 字符串string Docker 映像的 URL。URL for the Docker image.
basic_authbasic_auth DockerBasicAuthDockerBasicAuth Docker 存储库的基本身份验证信息。Basic authentication information for Docker repository.

DockerBasicAuthDockerBasicAuth

Docker 存储库基本身份验证信息。Docker repository basic authentication information.

字段Field 描述Description
usernameusername Docker 存储库的用户名。User name for the Docker repository.
passwordpassword Docker 存储库的密码。Password for the Docker repository.

LogSyncStatus LogSyncStatus

日志传送状态。Log delivery status.

字段名称Field Name 类型Type 描述Description
last_attemptedlast_attempted INT64 上次尝试的时间戳。The timestamp of last attempt. 如果上次尝试失败,last_exception 会包含上次尝试中的异常。If the last attempt fails, last_exception contains the exception in the last attempt.
last_exceptionlast_exception STRING 上次尝试引发的异常,如果上次尝试时没有异常,则该字段为 NULL(在响应中省略)。The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted.

NodeType NodeType

Spark 节点类型的说明,包括节点的维度以及将会在其上托管该节点的实例类型。Description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

字段名称Field Name 类型Type 描述Description
node_type_idnode_type_id STRING 此节点类型的唯一标识符。Unique identifier for this node type. 此字段为必需字段。This field is required.
memory_mbmemory_mb INT32 可用于此节点类型的内存(以 MB 为单位)。Memory (in MB) available for this node type. 此字段为必需字段。This field is required.
num_coresnum_cores FLOAT 可用于此节点类型的 CPU 内核数。Number of CPU cores available for this node type. 如果计算机实例上的内核数无法被该计算机上 Spark 节点数整除,则此数值可以是小数。This can be fractional if the the number of cores on a machine instance is not divisible by the number of Spark nodes on that machine. 此字段为必需字段。This field is required.
descriptiondescription STRING 与此节点类型关联的字符串说明。A string description associated with this node type. 此字段为必需字段。This field is required.
instance_type_idinstance_type_id STRING 运行此节点的硬件类型的标识符。An identifier for the type of hardware that this node runs on. 此字段为必需字段。This field is required.
is_deprecatedis_deprecated BOOL 该节点类型是否为已弃用。Whether the node type is deprecated. 未弃用的节点类型可提供更高的性能。Non-deprecated node types offer greater performance.
node_infonode_info ClusterCloudProviderNodeInfoClusterCloudProviderNodeInfo 云服务提供商报告的节点类型信息。Node type info reported by the cloud provider.

ClusterCloudProviderNodeInfoClusterCloudProviderNodeInfo

有关云服务提供商所提供实例的信息。Information about an instance supplied by a cloud provider.

字段名称Field Name 类型Type 说明Description
statusstatus ClusterCloudProviderNodeStatusClusterCloudProviderNodeStatus 由云服务提供商报告的状态。Status as reported by the cloud provider.
available_core_quotaavailable_core_quota INT32 可用的 CPU 内核配额。Available CPU core quota.
total_core_quotatotal_core_quota INT32 CPU 内核配额总量。Total CPU core quota.

ClusterCloudProviderNodeStatusClusterCloudProviderNodeStatus

云服务提供商所提供实例的状态。Status of an instance supplied by a cloud provider.

状态Status 描述Description
NotEnabledOnSubscriptionNotEnabledOnSubscription 不可用于订阅的节点类型。Node type not available for subscription.
NotAvailableInRegionNotAvailableInRegion 区域中未提供的节点类型。Node type not available in region.

ParameterPair ParameterPair

提供有关群集终止原因的更多信息的参数。Parameter that provides additional information about why a cluster was terminated.

类型Type 描述Description
TerminationParameterTerminationParameter 终止信息的类型。Type of termination information.
STRING 终止信息。The termination information.

SparkConfPair SparkConfPair

Spark 配置键值对。Spark configuration key-value pairs.

类型Type 描述Description
STRING 配置属性名称。A configuration property name.
STRING 配置属性值。The configuration property value.

SparkEnvPair SparkEnvPair

Spark 环境变量键值对。Spark environment variable key-value pairs.

重要

在作业群集中指定环境变量时,此数据结构中的字段只接受拉丁字符(ASCII 字符集)。When specifying environment variables in a job cluster, the fields in this data structure accept only Latin characters (ASCII character set). 使用非 ASCII 字符将会返回错误。Using non-ASCII characters will return an error. 例如,中文、日文汉字和表情符号都属于无效的非 ASCII 字符。Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis.

类型Type 描述Description
STRING 环境变量名称。An environment variable name.
STRING 环境变量值。The environment variable value.

SparkNode SparkNode

Spark 驱动程序或执行程序配置。Spark driver or executor configuration.

字段名称Field Name 类型Type 描述Description
private_ipprivate_ip STRING Spark 节点的专用 IP 地址(通常为 10.x.x.x 地址)。Private IP address (typically a 10.x.x.x address) of the Spark node. 此地址不同于主机实例的专用 IP 地址。This is different from the private IP address of the host instance.
public_dnspublic_dns STRING 此节点的公共 DNS 地址。Public DNS address of this node. 此地址可用于访问驱动程序节点上的 Spark JDBC 服务器。This address can be used to access the Spark JDBC server on the driver node.
node_idnode_id STRING 此节点的全局唯一标识符。Globally unique identifier for this node.
instance_idinstance_id STRING 来自云服务提供商的主机实例的全局唯一标识符。Globally unique identifier for the host instance from the cloud provider.
start_timestampstart_timestamp INT64 Spark 节点启动时的时间戳(以毫秒表示)。The timestamp (in millisecond) when the Spark node is launched.
host_private_iphost_private_ip STRING 主机实例的专用 IP 地址。The private IP address of the host instance.

SparkVersion SparkVersion

群集的 Databricks Runtime 版本。Databricks Runtime version of the cluster.

字段名称Field Name 类型Type 描述Description
keykey STRING Databricks Runtime 版本键,例如 7.3.x-scala2.12Databricks Runtime version key, for example 7.3.x-scala2.12. 在创建新群集时应以 spark_version 形式提供该值。The value that should be provided as the spark_version when creating a new cluster. 由于“通配符”版本(即,7.3.x-scala2.12 是一个“通配符”版本)存在小 bug 修复,确切的运行时版本可能会随时间的推移而更改。The exact runtime version may change over time for a “wildcard” version (that is, 7.3.x-scala2.12 is a “wildcard” version) with minor bug fixes.
namename STRING 运行时版本的描述性名称,例如 "Databricks Runtime 7.3 LTS"。A descriptive name for the runtime version, for example “Databricks Runtime 7.3 LTS”.

TerminationReason TerminationReason

终止群集的原因。Reason why a cluster was terminated.

字段名称Field Name 类型Type 说明Description
codecode TerminationCodeTerminationCode 指示群集终止原因的状态代码。Status code indicating why a cluster was terminated.
typetype TerminationTypeTerminationType 指示为何终止群集的原因。Reason indicating why a cluster was terminated.
parametersparameters ParameterPairParameterPair 对象,其中包含一组提供群集终止原因相关信息的参数。Object containing a set of parameters that provide information about why a cluster was terminated.

PoolClusterTerminationCode PoolClusterTerminationCode

状态代码,用于指示群集因池故障而终止的原因。Status code indicating why the cluster was terminated due to a pool failure.

代码Code 说明Description
INSTANCE_POOL_MAX_CAPACITY_FAILUREINSTANCE_POOL_MAX_CAPACITY_FAILURE 已达到池的最大容量。The pool max capacity has been reached.
INSTANCE_POOL_NOT_FOUND_FAILUREINSTANCE_POOL_NOT_FOUND_FAILURE 群集指定的池不再处于活动状态或不存在。The pool specified by the cluster is no longer active or doesn’t exist.

ClusterSource ClusterSource

创建该群集的服务。Service that created the cluster.

服务Service 说明Description
UIUI 通过 UI 创建的群集。Cluster created through the UI.
JOBJOB Databricks 作业计划程序创建的群集。Cluster created by the Databricks job scheduler.
APIAPI 通过 API 调用创建的群集。Cluster created through an API call.

ClusterState ClusterState

群集的状态。State of a cluster. 可允许的状态转换如下:The allowable state transitions are as follows:

  • PENDING -> RUNNING
  • PENDING -> TERMINATING
  • RUNNING -> RESIZING
  • RUNNING -> RESTARTING
  • RUNNING -> TERMINATING
  • RESTARTING -> RUNNING
  • RESTARTING -> TERMINATING
  • RESIZING -> RUNNING
  • RESIZING -> TERMINATING
  • TERMINATING -> TERMINATED
状态State 描述Description
PENDING 指示群集正处于创建过程中。Indicates that a cluster is in the process of being created.
RUNNING 指示群集已启动并已可供使用。Indicates that a cluster has been started and is ready for use.
RESTARTING 指示群集正处于重启过程中。Indicates that a cluster is in the process of restarting.
RESIZING 指示群集正处于添加或删除节点的过程中。Indicates that a cluster is in the process of adding or removing nodes.
TERMINATING 指示群集正处于销毁过程中。Indicates that a cluster is in the process of being destroyed.
TERMINATED 指示已成功销毁群集。Indicates that a cluster has been successfully destroyed.
ERROR 不会再使用此状态。This state is no longer used. 此状态曾用于指示未能创建的群集。It was used to indicate a cluster that failed to be created.
现已改用 TERMINATINGTERMINATEDTERMINATING and TERMINATED are used instead.
UNKNOWN 指示群集处于未知状态。Indicates that a cluster is in an unknown state. 群集永不应处于此状态。A cluster should never be in this state.

TerminationCode TerminationCode

指示该群集终止原因的状态代码。Status code indicating why the cluster was terminated.

代码Code 说明Description
USER_REQUESTUSER_REQUEST 用户已直接终止该群集。A user terminated the cluster directly. 参数应包括一个 username 字段,用于指示终止该群集的特定用户。Parameters should include a username field that indicates the specific user who terminated the cluster.
JOB_FINISHEDJOB_FINISHED 群集由某个作业启动,并在该作业完成时终止。The cluster was launched by a job, and terminated when the job completed.
INACTIVITYINACTIVITY 群集由于处于空闲状态而被终止。The cluster was terminated since it was idle.
CLOUD_PROVIDER_SHUTDOWNCLOUD_PROVIDER_SHUTDOWN 云服务提供商已终止托管 Spark 驱动程序的实例。The instance that hosted the Spark driver was terminated by the cloud provider.
COMMUNICATION_LOSTCOMMUNICATION_LOST Azure Databricks 已丢失与驱动程序实例上服务的连接。Azure Databricks lost connection to services on the driver instance. 例如,在云网络基础结构中出现问题时,或者,在实例本身运行不正常时,就可能发生这种情况。For example, this can happen when problems arise in cloud networking infrastructure, or when the instance itself becomes unhealthy.
CLOUD_PROVIDER_LAUNCH_FAILURECLOUD_PROVIDER_LAUNCH_FAILURE Azure Databricks 在请求实例以启动群集时遇到了云服务提供商故障。Azure Databricks experienced a cloud provider failure when requesting instances to launch clusters.
SPARK_STARTUP_FAILURESPARK_STARTUP_FAILURE 该群集未能初始化。The cluster failed to initialize. 可能的原因包括未能为 Spark 创建环境或者在启动 Spark Master 和工作进程时出现问题。Possible reasons may include failure to create the environment for Spark or issues launching the Spark master and worker processes.
INVALID_ARGUMENTINVALID_ARGUMENT 无法启动该群集,因为用户指定了无效的参数。Cannot launch the cluster because the user specified an invalid argument. 例如,用户可能为该群集指定了无效的运行时版本。For example, the user might specify an invalid runtime version for the cluster.
UNEXPECTED_LAUNCH_FAILUREUNEXPECTED_LAUNCH_FAILURE 在启动此群集时,Azure Databricks 未能完成关键设置步骤,因而导致终止该群集。While launching this cluster, Azure Databricks failed to complete critical setup steps, terminating the cluster.
INTERNAL_ERRORINTERNAL_ERROR Azure Databricks 遇到了意外错误,该错误已强制终止正在运行的群集。Azure Databricks encountered an unexpected error that forced the running cluster to be terminated. 有关其他详细信息,请联系 Azure Databricks 支持人员。Contact Azure Databricks support for additional details.
SPARK_ERRORSPARK_ERROR Spark 驱动程序未能启动。The Spark driver failed to start. 可能的原因包括库不兼容以及初始化脚本损坏了 Spark 容器。Possible reasons may include incompatible libraries and initialization scripts that corrupted the Spark container.
METASTORE_COMPONENT_UNHEALTHYMETASTORE_COMPONENT_UNHEALTHY 群集未能启动,因为无法访问外部元存储。The cluster failed to start because the external metastore could not be reached. 请参阅排查故障Refer to Troubleshooting.
DBFS_COMPONENT_UNHEALTHYDBFS_COMPONENT_UNHEALTHY 群集未能启动,因为无法访问 Databricks 文件系统 (DBFS)。The cluster failed to start because Databricks File System (DBFS) could not be reached.
AZURE_RESOURCE_PROVIDER_THROTTLINGAZURE_RESOURCE_PROVIDER_THROTTLING Azure Databricks 已达到 Azure 资源提供程序请求限制。Azure Databricks reached the Azure Resource Provider request limit. 具体而言,是指对特定资源类型(计算、网络等)的 API 请求速率不能超过该限制。Specifically, the API request rate to the specific resource type (compute, network, etc.) can’t exceed the limit. 重试可能有助于解决此问题。Retry might help to resolve the issue. 有关详细信息,请参阅 https://docs.microsoft.com/azure/virtual-machines/troubleshooting/troubleshooting-throttling-errorsFor further information, seehttps://docs.microsoft.com/azure/virtual-machines/troubleshooting/troubleshooting-throttling-errors.
AZURE_RESOURCE_MANAGER_THROTTLINGAZURE_RESOURCE_MANAGER_THROTTLING Azure Databricks 已达到 Azure 资源管理器请求限制,这将会阻止 Azure SDK 向 Azure 资源管理器发出任何读取或写入请求。Azure Databricks reached the Azure Resource Manager request limit which will prevent the Azure SDK from issuing any read or write request to the Azure Resource Manager. 请求限制应用于各个订阅,每小时一次。The request limit is applied to each subscription every hour. 请于一小时后重试,或者,更改为较小的群集大小可能会有助于解决此问题。Retry after an hour or changing to a smaller cluster size might help to resolve the issue. 有关详细信息,请参阅 https://docs.microsoft.com/azure/azure-resource-manager/resource-manager-request-limitsFor further information, seehttps://docs.microsoft.com/azure/azure-resource-manager/resource-manager-request-limits.
NETWORK_CONFIGURATION_FAILURENETWORK_CONFIGURATION_FAILURE 由于网络配置中的错误,已终止该群集。The cluster was terminated due to an error in the network configuration. 例如,具有 VNet 注入的工作区的 DNS 设置不正确,从而阻止了对工作器项目的访问。For example, a workspace with VNet injection had incorrect DNS settings that blocked access to worker artifacts.
DRIVER_UNREACHABLEDRIVER_UNREACHABLE Azure Databricks 无法访问 Spark 驱动程序,因为该驱动程序不可访问。Azure Databricks was not able to access the Spark driver, because it was not reachable.
DRIVER_UNRESPONSIVEDRIVER_UNRESPONSIVE Azure Databricks 无法访问 Spark 驱动程序,因为该驱动程序无响应。Azure Databricks was not able to access the Spark driver, because it was unresponsive.
INSTANCE_UNREACHABLEINSTANCE_UNREACHABLE Azure Databricks 无法访问实例,因而无法启动群集。Azure Databricks was not able to access instances in order to start the cluster. 这可能是暂时性的网络问题。This can be a transient networking issue. 如果问题持续存在,通常表明网络环境配置有误。If the problem persists, this usually indicates a networking environment misconfiguration.
CONTAINER_LAUNCH_FAILURECONTAINER_LAUNCH_FAILURE Azure Databricks 无法在群集的工作器节点上启动容器。Azure Databricks was unable to launch containers on worker nodes for the cluster. 请让管理员检查网络配置。Have your admin check your network configuration.
INSTANCE_POOL_CLUSTER_FAILUREINSTANCE_POOL_CLUSTER_FAILURE 池支持的群集特定的故障。Pool backed cluster specific failure. 有关详细信息,请参阅Refer to Pools for details.
REQUEST_REJECTEDREQUEST_REJECTED Azure Databricks 此时无法处理该请求。Azure Databricks cannot handle the request at this moment. 请稍后重试,如果问题仍然存在,请联系 Azure Databricks。Try again later and contact Azure Databricks if the problem persists.
INIT_SCRIPT_FAILUREINIT_SCRIPT_FAILURE Azure Databricks 无法在群集的某一节点上加载并运行群集范围的初始化脚本,或者该初始化脚本终止时返回了非零退出代码。Azure Databricks cannot load and run a cluster-scoped init script on one of the cluster’s nodes, or the init script terminates with a non-zero exit code. 请参阅初始化脚本日志Refer to Init script logs.
TRIAL_EXPIREDTRIAL_EXPIRED Azure Databricks 试用订阅已过期。The Azure Databricks trial subscription expired.

TerminationType TerminationType

群集的终止原因。Reason why the cluster was terminated.

类型Type 描述Description
成功SUCCESS 终止已成功。Termination succeeded.
CLIENT_ERRORCLIENT_ERROR 不可重试。Non-retriable. 重新尝试创建群集之前,客户端必须先修复参数。Client must fix parameters before reattempting the cluster creation.
SERVICE_FAULTSERVICE_FAULT Azure Databricks 服务问题。Azure Databricks service issue. 客户端可以重试。Client can retry.
CLOUD_FAILURECLOUD_FAILURE 云服务提供商基础结构问题。Cloud provider infrastructure issue. 客户端可以在基础问题得到解决之后重试。Client can retry after the underlying issue is resolved.

TerminationParameter TerminationParameter

提供群集终止原因其他相关信息的键。Key that provides additional information about why a cluster was terminated.

Key 描述Description
usernameusername 终止群集的用户的用户名。The username of the user who terminated the cluster.
databricks_error_messagedatabricks_error_message 可能说明群集终止原因的其他上下文。Additional context that may explain the reason for cluster termination.
inactivity_duration_mininactivity_duration_min 空闲群集在处于非活动状态的时间已达到此持续时间后关闭。An idle cluster was shut down after being inactive for this duration.
instance_idinstance_id 承载 Spark 驱动程序的实例的 ID。The ID of the instance that was hosting the Spark driver.
azure_error_codeazure_error_code Azure 提供了错误代码,该代码描述了无法预配群集节点的原因。The Azure provided error code describing why cluster nodes could not be provisioned. 有关参考信息,请参阅:https://docs.microsoft.com/azure/virtual-machines/windows/error-messagesFor reference, see: https://docs.microsoft.com/azure/virtual-machines/windows/error-messages.
azure_error_messageazure_error_message Azure 中各种不同故障的上下文,可供用户阅读。Human-readable context of various failures from Azure. 此字段是非结构化的,且其确切格式随时可能发生变更。This field is unstructured, and its exact format is subject to change.
instance_pool_idinstance_pool_id 群集正在使用的实例池的 ID。The ID of the instance pool the cluster is using.
instance_pool_error_codeinstance_pool_error_code 特定于某个池的群集故障的错误代码The error code for cluster failures specific to a pool.