question

arkiboys avatar image
0 Votes"
arkiboys asked HimanshuSinha-MSFT commented

sink delta parquet file rows duplicated on each run

When I debug step by step each of the transformations in dataflow, I see the correct results for:
sinkDeltaDelete
sinkDeltaInsert
sinkDeltaUpdate
140445-image.png
Question:
When I run the pipeline which has this dataflow in it, I get duplicated rows for the existing rows in the sink parquet file.

Any suggestions?

Thank you


azure-data-factory
image.png (51.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

Hello @arkiboys ,
Thanks for the ask and using Microsoft Q&A platform .
I see three flows running a parallel , may be thats the reason .

140704-image.png

Please do let me know how it goes .
Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (103.3 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,
So what do you suggest I change the design to?
Thanks

0 Votes 0 ·

Hello @arkiboys ,

Thanks for the ask again , Well since i have not seen the underlining data , i am not be very confident suggesting that , but then you can do the debug and see how the different branches behave and may be tweak or re-design the implementation .

Thanks
Himanshu

0 Votes 0 ·