question

Jeeva-6453 avatar image
0 Votes"
Jeeva-6453 asked AxelVulsteke-1308 answered

Caused by: com.databricks.NotebookExecutionException: FAILED

We have Databricks pipeline build and running using datafactory in batches with multiple files. At random, processing of a given file will fail causing the automation to abort processing that file and move on to the next. The error is consistent but indescript: “Caused by: com.databricks.NotebookExecutionException: FAILED” and the troubleshooting documentation for the error code thrown (3204) says “N/A” and applies to issues such as “an unexpected cluster state”. When we rerun the files through the same orchestration and pipelines or manually they typically work fine

Do you have any recommendations for troubleshooting this issue? Trace from DF.

DataBricks Stack Trace from DataFactory Output URL:
at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:75)
at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:122)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-418432851273673:1)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-418432851273673:46)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw.<init>(command-418432851273673:48)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw.<init>(command-418432851273673:50)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw.<init>(command-418432851273673:52)
at line1ee345426a1546219668c06720be0c2627.$read$$iw.<init>(command-418432851273673:54)
at line1ee345426a1546219668c06720be0c2627.$read.<init>(command-418432851273673:56)
at line1ee345426a1546219668c06720be0c2627.$read$.<init>(command-418432851273673:60)
at line1ee345426a1546219668c06720be0c2627.$read$.<clinit>(command-418432851273673)
at line1ee345426a1546219668c06720be0c2627.$eval$.$print$lzycompute(<notebook>:7)
at line1ee345426a1546219668c06720be0c2627.$eval$.$print(<notebook>:6)
at line1ee345426a1546219668c06720be0c2627.$eval.$print(<notebook>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:219)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:204)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:773)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:726)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:204)
at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$10(DriverLocal.scala:431)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:48)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:48)
at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:408)
at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:653)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:645)
at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:486)
at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:598)
at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:391)
at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.databricks.NotebookExecutionException: FAILED
at com.databricks.workflow.WorkflowDriver.run0(WorkflowDriver.scala:118)
at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:70)
at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:122)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-418432851273673:1)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-418432851273673:46)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw$$iw.<init>(command-418432851273673:48)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw$$iw.<init>(command-418432851273673:50)
at line1ee345426a1546219668c06720be0c2627.$read$$iw$$iw.<init>(command-418432851273673:52)
at line1ee345426a1546219668c06720be0c2627.$read$$iw.<init>(command-418432851273673:54)
at line1ee345426a1546219668c06720be0c2627.$read.<init>(command-418432851273673:56)
at line1ee345426a1546219668c06720be0c2627.$read$.<init>(command-418432851273673:60)
at line1ee345426a1546219668c06720be0c2627.$read$.<clinit>(command-418432851273673)
at line1ee345426a1546219668c06720be0c2627.$eval$.$print$lzycompute(<notebook>:7)
at line1ee345426a1546219668c06720be0c2627.$eval$.$print(<notebook>:6)
at line1ee345426a1546219668c06720be0c2627.$eval.$print(<notebook>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:219)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:204)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:773)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:726)
at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:204)
at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$10(DriverLocal.scala:431)
at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:48)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:48)
at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:408)
at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:653)
at scala.util.Try$.apply(Try.scala:213)
at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:645)
at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:486)
at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:598)
at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:391)
at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
at java.lang.Thread.run(Thread.java:748)

azure-data-factoryazure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered Jeeva-6453 commented

Hello @Jeeva-6453,

I would recommend you to enable retry in the activity using the below option and check if that works.

39887-image.png

Hope this helps. Do let us know if you any further queries.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.



image.png (5.1 KiB)
image.png (111.8 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Jeeva-6453,

Just checking in to see if the above answer helped. If this answers your query, do click “Accept Answer” and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @Jeeva-6453,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Take care & stay safe!

0 Votes 0 ·

Thanks. This is one of the viable solution fixes our issue.

0 Votes 0 ·
AxelVulsteke-1308 avatar image
0 Votes"
AxelVulsteke-1308 answered

Hi,
We do exact the same ETL: using Azure Data Factory to orchestrate Python Databricks notebooks and we experience the same behavior.

The retry policy solves the issue, but is there no better way to solve this?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.