question

AdminKandalaBhargav-9061 avatar image
0 Votes"
AdminKandalaBhargav-9061 asked KranthiPakala-MSFT commented

Why does not COPY actiivty automatically create file in ADLS Gen2

Hi,
I am creating a pipeline to COPY multiple files from my local network to ADLS, why does not COPY create files , why should i manually create the blank files in the storage explorer manually and then create dataset? Why cannot i just point to a folder on the storage explorer and the COPY Activity automatically copy all the files to the folder on the storage explorer?
Thanks

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If you point to a folder in Target dataset, ideally, copy activity should copy all files in that folder. What happens if you select a folder in dataset? Are you getting any error? Can you share few screenshots.

0 Votes 0 ·

@VaibhavChaudhari thanks. In the pipe line at the SINK stag when i try to create dataset and point to the TARGET folder, i get a message to specify the file and it wont let me click on the MAPPINGS tab (it is greyed out).
Thanks

0 Votes 0 ·
VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered KranthiPakala-MSFT commented

Are you creating Binary type of dataset as Source & Sink? I created the same for source and target one as below which points to temp folder

122981-image.png


and could run the copy activity - where I am copying files from one folder to another folder within same storage account

122983-image.png




Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav


image.png (18.0 KiB)
image.png (22.4 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@VaibhavChaudhari thanks and apologies for the delay. Here is the scenario.

1) Source is a .csv file (50 plus files), for explaining i am just considering only one file for now.
2) SFTP Server.
3) Created a Linked Service (connecting to the SFTP Server) , created another Linked Service to connect to ADLS Gen2 folder (Same IR used).
3) Created Source data set.
4) Created pipe line with Copy .
5) Pipe line failed with error "error at sink level, sink file not named"
Thanks

0 Votes 0 ·

Hi @AdminKandalaBhargav-9061,

Sorry for the delayed response. To better assist on this issue, could you please share the screenshot of your copy activity sink settings as well the sink dataset configuration? Also it would be great if you could share the failed pipeline runID, failed activity runID and complete error message you are receiving?

Since you are trying to copying multiple files from SFTP to ADLS Gen2, you can also utilize copyBehavior settings in your Copy sink.


Thank you


0 Votes 0 ·

Hi,

We still have not heard back from you. Just wanted to check if you are still facing the issue or need assistance on this? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue. Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

0 Votes 0 ·
KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered

Hi @AdminKandalaBhargav-9061,

Could you please confirm if your source and sink are Binary datasets? If yes, then you won't be able to do mapping and it will be greyed out as Mapping is not supported in Binary datasets.

Please note that when using Binary dataset, ADF does not parse file content but treat it as-is. Also when using Binary dataset in copy activity, you can only copy from Binary dataset to Binary dataset.

If you just want to copy files from source to sink location, you don't have to provide any file name in sink dataset, you just have to point the source files in source dataset and point your sink dataset to the folder where you would want to copy them. And the files should copy as is.

Hope this helps. Do let us know if you encounter any issues.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.