question

CowleEddie-4432 avatar image
0 Votes"
CowleEddie-4432 asked PRADEEPCHEEKATLA-MSFT commented

Error copying to SFTP sub folder uisng Azure copy activity

Hi,

I'm attempting to copy a file from Azure file storage to a remote SFTP location using the Azure Copy Activity.
The sink dataset in the copy is using an SFTP linked service where the authentication is SSH Public Key.
When I run the activity it successfully copies the file to the root of the SFTP location.

However, when I try and copy to a sub directory on the SFTP location it fails with the error:

 Failure happened on 'Sink' side. ErrorCode=SftpPathNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Can't find SFTP path '/OpenWork/test_report.txt'. Please check if the path exists. If the path you configured does not start with '/', note it is a relative path under the given user's default folder ''.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=Renci.SshNet.Common.SftpPathNotFoundException,Message=ObjectStore not found. Name: OpenWork,Source=Renci.SshNet,'

I know that the folder exists as I can see it using Filezilla (using the same authentication key). Using Filezilla I can also copy files to it, view the folder contents etc.

I've also noticed that within the Azure sink dataset, I am unable to Browse to the SFTP folder (neither the root nor the sub folder). In the case of browsing to the root I get:

 Permission denied to access '/'. Access denied 

Any idea why I can write to the root of the SFTP folder, but can't read from it, nor copy to the SFTP sub folder?

Thanks for any help

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @CowleEddie-4432,

Welcome to the Microsoft Q&A platform.

If you receive the error "UserErrorSftpPathNotFound," "UserErrorSftpPermissionDenied," or "SftpOperationFail" when you're writing data into SFTP, and the SFTP user you use does have the proper permissions, check to see whether your SFTP server support file rename operation is working. If it isn't, disable the Upload with temp file (useTempFileRename) option and try again. To learn more about this property, see the preceding table. If you use a self-hosted integration runtime for the Copy activity, be sure to use version 4.6 or later.

The following properties are supported for SFTP under storeSettings settings in a format-based Copy sink:

109825-image.png

Make sure the file list path in the Copy activity source as shown below:

109841-image.png

For more details, refer Copy data from and to the SFTP server by using Azure Data Factory.

Hope this helps. Do let us know if you any further queries.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.


image.png (75.1 KiB)
image.png (29.8 KiB)
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @CowleEddie-4432,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi,

Thanks for your response. I have already used the settings you have indicated. They are set as follows for the sink:

    "sink": {
             "type": "DelimitedTextSink",
             "storeSettings": {
                 "type": "SftpWriteSettings",
                 "operationTimeout": "01:00:00",
                 "useTempFileRename": false
             },
             "formatSettings": {
                 "type": "DelimitedTextWriteSettings",
                 "quoteAllText": true,
                 "fileExtension": ".txt"
             }
         },


Just to reiterate, I can succcesfully copy the file to the root of the target SFTP folder. It is copying to a subfolder on the target SFTP that fails.
So for example these setting work fine and copy is successful:

110293-image.png

But if I use these settings it fails:

110294-image.png

As mentioned, I am unable to use the 'Browse' button in both setups.

Any help would be appreciated!

0 Votes 0 ·
image.png (30.5 KiB)
image.png (29.1 KiB)

Hello @CowleEddie-4432,

Could you please confirm whether the username you used to authenticate the linked service does have the proper permissions on the subfolder on the target SFTP?

0 Votes 0 ·

Hi @PRADEEPCHEEKATLA-MSFT

Yes, the username does have correct permissions.

I know this because I am able to drop files to the subfolder using the same username but a different SFTP client (e.g. I can use Filezilla successfully).

Is there any other azure configuration I can share with you that might help?

Many thanks

0 Votes 0 ·

Hello @CowleEddie-4432,

This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.

0 Votes 0 ·

Hello @CowleEddie-4432,

Did you get a chance to open a support ticket?

0 Votes 0 ·