SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:

Rajaniesh Kaushikk 466 Reputation points MVP
2020-06-14T03:55:51.703+00:00

Hi,

I am running this code but this is throwing this error:

SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
0 comments No comments
{count} votes

Accepted answer
  1. Rajaniesh Kaushikk 466 Reputation points MVP
    2020-06-15T14:55:31.37+00:00

    Hi
    I was able to resolve this issue by adding attribute to the class for each class UID number will be different.

    @SerialVersionUID(114L)
    class Functions() extends Serializable {

    Regards
    Rajaniesh


1 additional answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 75,131 Reputation points Microsoft Employee
    2020-06-15T09:20:24.537+00:00

    @Rajaniesh Kaushikk ,

    The above error can be triggered when you initialize a variable on the driver (master), but then try to use it on one of the workers. In that case, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable.

    For more details, refer “Job aborted due to stage failure: Task not serializable:”.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.