sink delta parquet file rows duplicated on each run

Question

question

arkiboys asked Oct 14, '21 HimanshuSinha-MSFT commented Oct 21, '21

sink delta parquet file rows duplicated on each run

When I debug step by step each of the transformations in dataflow, I see the correct results for:
sinkDeltaDelete
sinkDeltaInsert
sinkDeltaUpdate

Question:
When I run the pipeline which has this dataflow in it, I get duplicated rows for the existing rows in the sink parquet file.

Any suggestions?

Thank you

azure-data-factory

image.png (51.6 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-10-14T21:02:59.363Z

HimanshuSinha-MSFT answered Oct 14, '21 HimanshuSinha-MSFT commented Oct 21, '21

Hello @arkiboys ,
Thanks for the ask and using Microsoft Q&A platform .
I see three flows running a parallel , may be thats the reason .

Please do let me know how it goes .
Thanks
Himanshu

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

image.png (103.3 KiB)

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

arkiboys · Oct 15, 2021 at 03:05 AM

Hi,
So what do you suggest I change the design to?
Thanks

0 ·

HimanshuSinha-MSFT arkiboys · Oct 21, 2021 at 06:58 PM

Hello @arkiboys ,

Thanks for the ask again , Well since i have not seen the underlining data , i am not be very confident suggesting that , but then you can do the debug and see how the different branches behave and may be tweak or re-design the implementation .

Thanks
Himanshu

0 ·

question