question

Kira-22 avatar image
0 Votes"
Kira-22 asked ShaikMaheer-MSFT commented

Configuring sink data set in azure data factory

0


I am trying to copy multiple folders with their files (.dat and .csv ) from ftp to Azure storage account , so I am using a get metadata for each and copy activity. My problem is that when setting the file path in the output data set I am not sure how to set the filename so it picks up all files in my folder.

I added a filename parameter in the data set and in the copydata sink I set it as @item().name but it's not working instead of copying the files, it copies the folder. the second try is that I dont set the filename in the directory, and it does copy the files but it adds the extension.txt to the files instead of keeping their original format.

Thank you for your help

azure-data-factoryazure-storage-accounts
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Kira-22 ,

Following up to check is below provided answer helps you? If yes, Please consider accepting answer by clicking on 130250-acceptansw.jpg button. Please let us know if any further queries. Thank you.

0 Votes 0 ·
acceptansw.jpg (2.5 KiB)

Hi @Kira-22 ,

Just checking is below provided answer helps you? If yes, Please consider accepting answer. Please let us know if any further queries. Thank you.

0 Votes 0 ·

Hi @Kira-22 ,

Could you please confirm is below provided answer helps you? If yes, Please consider accepting answer. Please let us know if any further queries. Thank you.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @Kira-22 ,

Thank you for posting query in Microsoft Q&A Platform.

It seems your GetMetaData activity is giving you Folders names array and you are passing that folder names array to ForEach activity to iterate over each folder.

So, here inside Copy Activity source tab use "wildcard file path" and use patterns to copy .csv and .dat files as shown in below screenshot.
127816-image.png

To store files in desired extension you can leverage Copy activity --> Sink tab --> File extension field
127832-image.png


Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.



image.png (285.0 KiB)
image.png (104.6 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi ,

The solution works but partially , for files with .DAT extension instead of storing the files in the desired extension which is .dat it adds .dat to the already existing extension so instead of File.DAT the result is File.DAT.DAT in the output

0 Votes 0 ·

Hi @Kira-22 ,

Thank you for response. To aviod issue with extensions please consider using Binary Format dataset for both cases(.DAT & .CSV). Binary format dataset helps to copy file as is without touching files extension or data. Please check below image for more details. I tried in my repro with Binary format dataset. Its working fine.

131792-image.png

Hope this will help. Thank you.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer.

0 Votes 0 ·
image.png (425.8 KiB)