Datafactory dataflow run on Azure IR With Time to Live(TTL>0) Error

Question

question

lsc2021 asked · Mar 10 at 02:19 AM

Datafactory dataflow run on Azure IR With Time to Live(TTL>0) Error

When I Run PipeLine including dataflow on Azure IR (TTL = 15'), the following error occurred.
.error message: Operation on target'dataflow_test' failed: Operation returned an invalid status code'InternalServerError'

.dataflow [dataflow_test] is simple:
Source (csv)-> DerivedColumn-> sink (csv)
*source, sink on azure IR (TTL = 0)

.pipeline's dataflow Setting:
1. Run on (Azure IR): Azure IR With Time to Live (TTL) 15 minutes -> run error
2. Run on (Azure IR): Azure IR With Time to Live (TTL) 0 minutes -> run success

If you know the solution, Please help me!

azure-data-factory

· 5

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT · Mar 10 at 06:46 AM

Hi @lsc2021,

Welcome to Microsoft Q&A forum and thanks for your query. As far as I have noticed InternalServerError occurs when there is an issue with ADF partner service (databricks).

Could you please check if this is a transient issue or are you experiencing this consistently? If you are experiencing this issue consistently, could you please share your failed pipeline runID, failed activity runID and the region of your Data factory so that we can check with internal team about the root cause and possible resolution/workaround? Also please provide the exact time when you have initially noticed this behavior.

We look forward to your response.

Thanks

0 ·

lsc2021 KranthiPakala-MSFT · Mar 10 at 07:06 AM

Thank you for your answer!

Run info:
.PipeLine RunID: 6501db66-c135-49a2-93ce-1421d6ef99fb
.dataflow activity runID: d4b3057b-8def-45f7-af34-3e1ce5bd541b
.region: japan east

Also please provide the exact time when you have initially noticed this behavior

->create and test azure IR(TTL>0) date: 2021/1/29

0 ·

KranthiPakala-MSFT lsc2021 · Mar 11 at 12:18 AM

Hi @lsc2021,

Could you please confirm if you are getting error every time TTL is set > 0 ? Are you still experiencing this issue consistently? Could you please try setting TTL with different time (eg: 20 min, 40 min) and let us know if this is consistently happening for TTL >0 .

We look forward to your confirmation.

Thanks

0 ·

Show more comments

Answer 1 · 2021-03-11T03:13:05.47Z

lsc2021 answered · Mar 11 at 03:13 AM

@KranthiPakala-MSFT
As requested, today I created 3 new IRs((10,15,30Min)) and test, all success!.
Thank you for your support.
We will close this QA.

· 1 ·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT · Mar 11 at 06:07 PM

Glad to know that the creating a new Azure IR resolved the issue. Thanks for confirming.

0 ·

Answer 2 · 2021-03-11T00:42:14.907Z

KranthiPakala-MSFT answered · Mar 11 at 12:42 AM

@lsc2021, Engineering team is looking into the issue based on the run ID's provided. In the meantime, to mitigate the issue, could you please try creating a new Azure IR with TTL enabled and check if you notice the same behavior or not? If you notice the same error even after creating a new Azure IR with TTL, please let us know so that we can update back to product team .

Thank you

·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question