question

arkiboys avatar image
0 Votes"
arkiboys asked arkiboys commented

delete row in parquet file - Delta

Hello,
Using data flow, In the Sink Delta parquet file, I would like to add an extra column called Deleted.
And set the default to be 0.
And then, whenever a row is deleted, I would like this field to be set to 1 for that very row.
How is this done please?
Thank you

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MarkKromer-MSFT avatar image
0 Votes"
MarkKromer-MSFT answered arkiboys commented

To add a new column with a default value per row, add a Derived Column, call the column as "Deleted" and set the value to 0.

Not sure what you mean by setting it to 1 when it is deleted? Unless I am missing something here, the row won't exist anymore when it is deleted.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,
If the row is deleted at source, then in the Delta sink parquet file I would like the deleted column to be set to 1 for that deleted row but 0 for all the other rows
Hope this helps
Thank you

0 Votes 0 ·

You will have to detect that the row was deleted by using an Exists transformation so that you can filter just the rows that were deleted in your data flow

0 Votes 0 ·
arkiboys avatar image arkiboys MarkKromer-2402 ·

I think what you mean is to leave the alter row as is with the upsert if as it works fine.
But for the delete bit to add the exists activity.
What do I do in exists to indicate a row is deleted and then do I need to use Delete If later in lter row?
Thank you

0 Votes 0 ·