Spark_Ambiguous_Executor_MaxExecutorFailures

Question

Hi

I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error

DAG = {

"activities": [

    {

        "name": "Notebook",

        "path": "Notebook1", 

        "timeoutPerCellInSeconds": 1800,

    },

    {

        "name": "Notebook2",

        "path": "Notebook2",

        "timeoutPerCellInSeconds": 1800

    },

    {

        "name": "Notebook3",

        "path": "Notebook3",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook4",

        "path": "Notebook4",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook5",

        "path": "Notebook5",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook6",

        "path": "Notebook6",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook7",

        "path": "Notebook7",

        "timeoutPerCellInSeconds": 1800

    },

],

"timeoutInSeconds": 43200,

"concurrency": 5

}

mssparkutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": False})

Error :

Spark_Ambiguous_Executor_MaxExecutorFailures: Livy session has failed. Session state: Dead. Error code: Spark_Ambiguous_Executor_MaxExecutorFailures. Job failed during run time with state=[dead].

TSG:Application failed because too many executors failed. The number of acceptable

executor failures is controlled by the config spark.yarn.max.executor.failures.

To investigate this failure, look at the executor logs and error codes.

Source: Unknown.

Answer 1

Hello Seelan,

The task failed because the executor that is running was lost.

This may happen because the task crashed the JVM.

Here are the Possible causes:

1. Driver memory issues
2. Executor Memory issues
3. Executor lost

Please check executor logs and error codes and see if you find any more details.