question

VictorYang-0370 avatar image
0 Votes"
VictorYang-0370 asked ShaikMaheer-MSFT edited

rowsWritten from data flow is always 0

I tried to get rowsWritten from data flow output log (with @activity('dataflowActivity').output.runStatus.metrics.sink1.rowsWritten.) , however it always show 0, the other number in the log looks good..108234-dataflow-output-vy.png


azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

just want to add, the data did inserted to target.

0 Votes 0 ·
ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT edited

Hi @VictorYang-0370 ,

Thank you for sharing details on requested clarifications.

When writing to a delta sink, there is a known limitation where the numbers of rows written won't be return in the monitoring output.

Kindly check below documentation for same(especially Known limitations section)
https://docs.microsoft.com/en-us/azure/data-factory/format-delta

Hope this will help.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT edited

Hi @VictorYang-0370 ,

Thank you for posting query in Microsoft Q&A Platform.
I ran a sample dataflow activity on source file with data of 2 rows. I could see "rowsWritten" property capturing number of rows copied correctly.
Also, below microsoft documention says "rowsWritten" property tells you number of rows actually copied to sink.
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-execute-data-flow-activity#monitoring-the-data-flow-activity

108579-image.png

Could you please re-check your execution once. May be try to delete all rows from Sink table and then execute Data flow to examine how many rows actually copied to Sink table and also what is value the value in "rowsWritten" property of data flow output json.

Hope this will help. Please let us know for any queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a email-notifications.


image.png (78.2 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Can you please change your sink target to delta format and see?

0 Votes 0 ·

Hi @VictorYang-0370 ,

Thank you for follow up query. Could you please help me on below clarification.

  • What type of dataset you are using in Sink transformation?

  • When you say change sink target to delta format, I am not getting that? It would be great help if you can share more details about it?

0 Votes 0 ·

the sink format is inline - delta format. it will create delta lake table in ADLS.

0 Votes 0 ·

I did another testing and found it seems only not working for delta format. I added csv file as another sink and I can see the rowwritten value for that sink, however for the delta sink, still show 0.

0 Votes 0 ·

yes @VictorYang-0370 . Its giving zero. Its known limitation. Please check New Answer posted below. Thank you

0 Votes 0 ·