question

DavidBeavon-2754 avatar image
0 Votes"
DavidBeavon-2754 asked SaurabhSharma-msft commented

Azure Spark Configuration (Environment) Documentation

In the environment tab of my cluster, there are lots of properties starting with "spark.databricks.clusterUsageTags".

However if I google for them I cannot find any information.

Is there any documentation or reference sheet for these values? Can someone please point me to it? Some of them are self evident and others are not.

I'm particularly interested in the last seven below.


spark.databricks.clusterUsageTags.autoTerminationMinutes
spark.databricks.clusterUsageTags.azureSubscriptionId
spark.databricks.clusterUsageTags.clusterAllTags
spark.databricks.clusterUsageTags.clusterAvailability
spark.databricks.clusterUsageTags.clusterCreator
spark.databricks.clusterUsageTags.clusterFirstOnDemand
spark.databricks.clusterUsageTags.clusterGeneration
spark.databricks.clusterUsageTags.clusterLogDeliveryEnabled
spark.databricks.clusterUsageTags.clusterLogDestination
spark.databricks.clusterUsageTags.clusterMaxWorkers
spark.databricks.clusterUsageTags.clusterMetastoreAccessType
spark.databricks.clusterUsageTags.clusterMinWorkers
spark.databricks.clusterUsageTags.clusterName
spark.databricks.clusterUsageTags.clusterTargetWorkers
spark.databricks.clusterUsageTags.clusterWorkers
spark.databricks.clusterUsageTags.containerType
spark.databricks.clusterUsageTags.dataPlaneRegion
spark.databricks.clusterUsageTags.driverContainerId
spark.databricks.clusterUsageTags.driverContainerPrivateIp
spark.databricks.clusterUsageTags.driverInstanceId
spark.databricks.clusterUsageTags.driverInstancePrivateIp
spark.databricks.clusterUsageTags.driverNodeType
spark.databricks.clusterUsageTags.driverPublicDns
spark.databricks.clusterUsageTags.region
spark.databricks.clusterUsageTags.sparkVersion
spark.databricks.clusterUsageTags.workerEnvironmentId

spark.databricks.clusterUsageTags.clusterNodeType
spark.databricks.clusterUsageTags.clusterState
spark.databricks.clusterUsageTags.clusterStateMessage
spark.databricks.clusterUsageTags.clusterId
spark.databricks.clusterUsageTags.cloudProvider
spark.databricks.clusterUsageTags.isSingleUserCluster
spark.databricks.clusterUsageTags.clusterNoDriverDaemon

azure-databricksdotnet-ml-big-data
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @DavidBeavon-2754,

Thanks for using Microsoft Q&A !!

Unfortunately, I do not see any documentation on these properties but I am checking internally if I can get any more information around the same. I will get back to you on the same.

Thanks
Saurabh

0 Votes 0 ·

@SaurabhSharma-msft In particular I'm hoping there is a way to create a "normal" cluster, that has multiple driver processes rather than one massive driver daemon.

In databricks this property always seems to be false:
spark.databricks.clusterUsageTags.clusterNoDriverDaemon

Perhaps there is a cluster type where databricks behaves normally. We are running into a bottleneck as a result of our IPC between the driver and external processes running on the driver node.

0 Votes 0 ·

@davidbeavon-2754 Thanks for sharing details. I am still waiting for products team response. I will provide the context to them as well.

Thanks
Saurabh

0 Votes 0 ·

0 Answers