question

UyenToBaoVitalifyAsia-9392 avatar image
0 Votes"
UyenToBaoVitalifyAsia-9392 asked ShaikMaheer-MSFT commented

Error when running dataflow

Dear MS Support,

I am Uyen.

We have an error in dataflow that first happened about 2 weeks ago and we could not find out any reasons or solution.
We really need your help.

We have a For Each activity and dataflows are executed inside. (20 dataflows in total)
Sometimes error happened in some of these dataflows but sometimes not.
Error message is as below.


Error code
DFExecutorUserError
Troubleshooting guide
Failure type
User configuration issue
Details
{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.","Details":"com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)\n\tat com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)\n\tat com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)\n\tat com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)\n\tat com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)\n\tat com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)\n\tat org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:64)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.a"}



Hope to hearing from you soon.
Thank you.

azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @UyenToBaoVitalifyAsia-9392 ,

Welcome to Microsoft Q&A Portal. Thank you for posting your query on Microsoft Q&A Platform.

  • So, you mean you have 20 data flow activities inside ForEach activity? Or Data flow inside For each activity is executing 20 times because of 20 iterations?

  • Did you tried using Sequential setting inside For each activity and check if that resolved issue?

Details on above clarifications may help to troubleshoot issue. Thank you.

1 Vote 1 ·

Hi ShaikMaheer

Thank you a lot for your response.

So, you mean you have 20 data flow activities inside ForEach activity? Or Data flow inside For each activity is executing 20 times because of 20 iterations?

=> We have only one data flow inside For each activity and it is executing 20 times because of 20 iterations.

D

id you tried using Sequential setting inside For each activity and check if that resolved issue?

=> We have not tried using Sequential yet because we need as much data flow runs parallel as possible. We have to process ~100GB data in a short time. If it is Sequential then it will not start next data flow unless the previous one is finished, right? Then the process time would be longer, I think.

Hope to hearing from you soon. Thank you.
0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT UyenToBaoVitalifyAsia-9392 ·

Hi @UyenToBaoVitalifyAsia-9392 ,

I did a research on this issue, it seems this kind of error may happen when your target or source data stores are not reachable due to some network latency or glitch.

Could you please try to make to make sure your source and sink data stores are online and then try to run and see. Also, If possible try using batch setting inside ForEach to define as 10 executions in parallel something like that and see how it behaves?

0 Votes 0 ·
Show more comments

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @UyenToBaoVitalifyAsia-9392 ,

Thank you for posting query on Microsoft Q&A Platform.

This kind of issue will come when you may have reached integration runtimes capacity limit or you might be running large amount of data flow using same integration runtime at same time.

You should consider running your pipeline data flow at different times or split your data flow execution's to across multiple integration runtimes.

Kindly check below thread where detailed resolution is discussed.
https://docs.microsoft.com/en-us/answers/questions/282303/azure-datafactory-error.html

Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @UyenToBaoVitalifyAsia-9392 ,

Following up to check is above answer helps you? If yes, Please Accept Answer. Accepting answer helps community too. Please let us know if any further queries. Thank you.

0 Votes 0 ·