Dataflow Activity giving output in different formats - "runStatus" vs. "SubStatus"

Question

question

RajasThakur-7366 asked Apr 29, '22 AnnuKumari-MSFT commented May 5, '22

Dataflow Activity giving output in different formats - "runStatus" vs. "SubStatus"

Hello,

We have some excel files, which are loaded into the Datalake as .txt files through CopyData Activity.
After that, We have a Data Flow activity to copy data from my DataLake to Azure SQL DB. We have implemented some logic after that for logging, for which we need to access the output of the Data Flow Activity. Ideally, we should get "runStatus" from the JSON output of the activity. But for some reason, we are getting "SubStatus" instead of "runStatus" in the output in some instances.

Does anyone know what is the reason for getting "SubStatus" instead of "runStatus"?
Is there any way in which we can make sure we always get "runStatus" from the DataFlow activity output?

Thanks

azure-data-factory

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RajasThakur-7366 · May 02 at 04:19 PM

Hello @AnnuKumari-MSFT,

Thank you for your reply.
Yes, your understanding is correct. We are running this DataFlow block inside a foreach block for multiple tables. We are expecting this output from the DataFlow Activity in each run:

For some instances, we are getting output like this:

Because we are getting "SubStatus" instead of "runStatus" in some cases, the blocks downstream fail causing the whole pipeline to fail. However, when we run the pipeline again for the specific file that caused this error, it goes through fine.

I checked the DataFlow activity in both instances (when getting JSON output as "SubStatus" and "runStatus") for input parameters it gets and they were the same. Also the DataFlow activity seems to be completed just fine (the data gets inserted in the tables),but the JSON output of that activity is different.

Thank you for your help!

[2]: /answers/storage/attachments/198220-image-2.png

1 ·

image-1.png (27.4 KiB)

image-2.png (37.4 KiB)

Answer 1 · 2022-05-02T10:17:24.18Z

AnnuKumari-MSFT answered May 2, '22 AnnuKumari-MSFT commented May 5, '22

Hi @RajasThakur-7366 ,

Welcome to Microsoft Q&A platform and thankyou for posting your query.

Going through your query, it looks like the JSON output of dataflow is giving you SubStatus as the property instead of runStatus Property. Please let me know if my understanding is incorrect.

Could you please share the screenshot of the same so that I can approach the internal product team to investigate further. I tried to reproduce your scenario but didn't face similar challenge. For me , dataflow output gave runStatus property. Thanks in advance !

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AnnuKumari-MSFT · May 04 at 07:52 AM

Hi @RajasThakur-7366 ,

Thankyou for providing detailed screenshot . I escalated this issue to the internal Product team and got to know they have recently deployed a hotfix for this issue. Could you please confirm if you are still experiencing the issue or has it been resolved at your end? Thanks in advance!

If the issue is resolved, please click Accept Answer as accepted answer helps community as well.

0 ·

RajasThakur-7366 AnnuKumari-MSFT · May 04 at 07:46 PM

Hello @AnnuKumari-MSFT ,

Thanks for the update. The last time we saw this issue was on April 29th. After that, we haven't encountered this issue so I believe that the hotfix that you have mentioned should have resolved the issue.

Thank you for your timely updates. I appreciate it.

1 ·

AnnuKumari-MSFT RajasThakur-7366 · May 05 at 04:25 AM

Hi @RajasThakur-7366 ,
Thanks for the confirmation !

0 ·

question