question

PratikAgrawalla-3442 avatar image
0 Votes"
PratikAgrawalla-3442 asked PratikAgrawalla-3442 commented

How can i append files from azure blob storage to a file present in file share using azure data factory ?

I want to use Azure data factory to append multiple files present in a blob container to a file present in file share. As the size of the file is big i want to go with Azure data factory .I am creating a data set of file system and have chosen binary format of data . I don't find any option to append into file share.

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If this answered your query, please mark as accepted answer. If not, please let me know how I may better assist.

0 Votes 0 ·

Thanks , you answered my query .

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

Hello @PratikAgrawalla-3442 and welcome to Microsoft Q&A.

I'm sorry to say, your ask is not possible. Please allow me to explain why.

Correctly appending data is not simply smushing one BLOB against another BLOB. Especially so with binary, as binary can be any file type, any content type, any encoding. Binary moves as-is without parsing or interpreting or any processsing. If binary could append, it would open the door for nonsensical combinations like appending sound to an image or executable.

Even for two of the same file type, it is more complicated.
Parquet has special sections that could be invalidated by the append.
Delimited text can have header rows, and a simple smush would place the second header in the middle of the resulting file, breaking the ability for downstream applications to ingest it.

Currently you cannot append data in blob storage.

The next best thing you could do, is place the 2 files next to each other, then do another copy activity using both of them together as source. In the settings you can merge the files.

The difference between merge and append, is that everything to combine is in source for merge. Append expects adding to, not overwriting the sink target. You would need to use something other than binary for this.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.