Apache Spark 作业失败,出现Failed to parse byte stringApache Spark job fails with Failed to parse byte string

问题Problem

Spark-提交作业失败,并出现 Failed to parse byte string: -1 错误消息。Spark-submit jobs fail with a Failed to parse byte string: -1 error message.

java.util.concurrent.ExecutionException: java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:182)
... 108 more
Caused by: java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1

原因Cause

spark.driver.maxResultSize应用程序属性的值为负。The value of the spark.driver.maxResultSize application property is negative.

解决方案Solution

分配的值,用于 spark.driver.maxResultSize 定义每个 Spark 操作的序列化结果的最大大小 ((以字节为单位)) 。The value assigned to spark.driver.maxResultSize defines the maximum size (in bytes) of the serialized results for each Spark action. 可以为属性分配一个正值, spark.driver.maxResultSize 以定义特定大小。You can assign a positive value to the spark.driver.maxResultSize property to define a specific size. 你还可以将值指定为0来定义无限大小的最大值。You can also assign a value of 0 to define an unlimited maximum size. 不能将负值分配给此属性。You cannot assign a negative value to this property.

如果作业的总大小大于 spark.driver.maxResultSize 值,则作业将中止。If the total size of a job is above the spark.driver.maxResultSize value, the job is aborted.

在设置过高的 (或不受限制的) 值时,应小心 spark.driver.maxResultSizeYou should be careful when setting an excessively high (or unlimited) value for spark.driver.maxResultSize. 如果 spark.driver.memory 未将属性设置得足够高,则高限制可能导致驱动程序中出现内存不足错误。A high limit can cause out-of-memory errors in the driver if the spark.driver.memory property is not set high enough.

有关更多详细信息,请参阅Spark 配置应用程序属性See Spark Configuration Application Properties for more details.