question

kuljeetpanag-0695 avatar image
0 Votes"
kuljeetpanag-0695 asked SumanthMarigowda-MSFT edited

How can we validate the destination file data with source table after running adf pipeline?

Is there any way to validate the parquet file data (row by row) from blob storage with source table?

azure-data-factoryazure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @kuljeetpanag-0695,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @kuljeetpanag-0695,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @kuljeetpanag-0695 ,
Thanks for the ask and using Microsoft Q&A platform .
I think one way to go is to use the Mapping data flow . In mapping data flow you can create two sources and then use a JOIN transformation with NOT EXIST option .


if you review the thread : https://docs.microsoft.com/en-us/answers/questions/578219/data-flow-34delete-if34-setting-in-alter-row.html?childToView=580571#answer-580571 , it does use the join and it should give you some idea .


138652-image.png


Please do let me know how it goes .
Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (165.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.