对高并发性群集的 Python 命令失败Python commands fail on high concurrency clusters

问题Problem

你正在尝试在高并发群集上运行 Python 命令。You are attempting to run Python commands on a high concurrency cluster.

所有 Python 命令均失败并出现 WARN 错误消息。All Python commands fail with a WARN error message.

WARN PythonDriverWrapper: Failed to start repl ReplId-61bef-9fc33-1f8f6-2
ExitCodeException exitCode=1: chown: invalid user: ‘spark-9fcdf4d2-045d-4f3b-9293-0f’

原因Cause

spark.databricks.pyspark.enableProcessIsolation truespark.databricks.session.share true 都是在群集上的 Apache Spark 配置中设置的。Both spark.databricks.pyspark.enableProcessIsolation true and spark.databricks.session.share true are set in the Apache Spark configuration on the cluster.

这两个 Spark 属性相互冲突,阻止群集运行 Python 命令。These two Spark properties conflict with each other and prevent the cluster from running Python commands.

解决方案Solution

你一次只能在群集上启用这两个 Spark 属性中的一个。You can only have one of these two Spark properties enabled on your cluster at a time.

必须根据需要选择 "进程隔离" 或 "Spark 共享会话"。You must choose process isolation or a Spark shared session based on your needs. 禁用另一个选项。Disable the other option.