question

lsc2021 avatar image
0 Votes"
lsc2021 asked ·

Datafactory dataflow run on Azure IR With Time to Live(TTL>0) Error

When I Run PipeLine including dataflow on Azure IR (TTL = 15'), the following error occurred.
.error message: Operation on target'dataflow_test' failed: Operation returned an invalid status code'InternalServerError'

.dataflow [dataflow_test] is simple:
Source (csv)-> DerivedColumn-> sink (csv)
*source, sink on azure IR (TTL = 0)

.pipeline's dataflow Setting:
1. Run on (Azure IR): Azure IR With Time to Live (TTL) 15 minutes -> run error
2. Run on (Azure IR): Azure IR With Time to Live (TTL) 0 minutes -> run success

If you know the solution, Please help me!

azure-data-factory
· 5
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @lsc2021,

Welcome to Microsoft Q&A forum and thanks for your query. As far as I have noticed InternalServerError occurs when there is an issue with ADF partner service (databricks).

Could you please check if this is a transient issue or are you experiencing this consistently? If you are experiencing this issue consistently, could you please share your failed pipeline runID, failed activity runID and the region of your Data factory so that we can check with internal team about the root cause and possible resolution/workaround? Also please provide the exact time when you have initially noticed this behavior.

We look forward to your response.

Thanks

0 Votes 0 ·
lsc2021 avatar image lsc2021 KranthiPakala-MSFT ·

Thank you for your answer!

Run info:
.PipeLine RunID: 6501db66-c135-49a2-93ce-1421d6ef99fb
.dataflow activity runID: d4b3057b-8def-45f7-af34-3e1ce5bd541b
.region: japan east

Also please provide the exact time when you have initially noticed this behavior

->create and test azure IR(TTL>0) date: 2021/1/29
0 Votes 0 ·

Hi @lsc2021,

Could you please confirm if you are getting error every time TTL is set > 0 ? Are you still experiencing this issue consistently? Could you please try setting TTL with different time (eg: 20 min, 40 min) and let us know if this is consistently happening for TTL >0 .

We look forward to your confirmation.

Thanks

0 Votes 0 ·
Show more comments
lsc2021 avatar image
0 Votes"
lsc2021 answered ·

@KranthiPakala-MSFT
As requested, today I created 3 new IRs((10,15,30Min)) and test, all success!.
Thank you for your support.
We will close this QA.

· 1 ·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Glad to know that the creating a new Azure IR resolved the issue. Thanks for confirming.

0 Votes 0 ·
KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered ·

@lsc2021, Engineering team is looking into the issue based on the run ID's provided. In the meantime, to mitigate the issue, could you please try creating a new Azure IR with TTL enabled and check if you notice the same behavior or not? If you notice the same error even after creating a new Azure IR with TTL, please let us know so that we can update back to product team .

Thank you

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.