question

DivyaRajendran-5109 avatar image
0 Votes"
DivyaRajendran-5109 asked HimanshuSinha-MSFT commented

ADF pipeline fails from ADLS gen2 to synapse

Hi ,

I have created a data flow from ADLS source to Synapse table. This dataflow has one alterrow with upsertif true()
Screenshots below show the settings I used in the dataflow and pipeline.

I am able to run this dataflow with under 1000 records without any pipeline failures.

However, I need to run this dataflow for 1million records. I always get the below errors, I could not find any good information on the documentation.
Could you please help!


Error 1: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: The connection is closed.","Details":"shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:234)\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1217)\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:3508)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:713)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:839)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:839)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:987)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:987)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:"}


Error 2. {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Could not find prepared statement with handle 1.","Details":"java.sql.BatchUpdateException: Could not find prepared statement with handle 1.\n\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2085)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:672)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:839)\n\tat org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:839)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:987)\n\tat org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:987)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2321)\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2321)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.doRunTask(Task.scala:140)\n\tat or"}


130860-screen-shot-2021-09-09-at-80052-pm.png
130913-screen-shot-2021-09-09-at-80119-pm.png

130837-screen-shot-2021-09-09-at-80216-pm.png

130867-screen-shot-2021-09-09-at-80301-pm.png





azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DivyaRajendran-5109 .,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @DivyaRajendran-5109 .,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered DivyaRajendran-5109 commented

Hello @DivyaRajendran-5109 .
Thanks for the ask and using the Microsoft Q&A platform .

Can you please try and set "Enable Staging " option and try if that helps ?

131668-image.png

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members





image.png (35.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@HimanshuSinha-MSFT Thank you for this solution.

It helped halfway, I had to increase the memory and compute on Synapse pool for this to work alongside checking this enable staging box.

0 Votes 0 ·