question

ZacharyLegault-8153 avatar image
0 Votes"
ZacharyLegault-8153 asked ZacharyLegault-8153 commented

AZCopy and S3 Authentication

We're trying to use AZCopy or MSE to transfer data from an S3 bucket into a storage blob but we're having issues getting AZCopy to authenticate with the S3 bucket. We get the following error:

failed to perform copy command due to error: s3 authentication to <s3 url>.s3-ap-southeast-2.amazonaws.com is not currently suported in AzCopy

We can't seem to find any resource that says exactly where S3 authentication is supported. Is there any info on how we can work around this/find out when this may be supported?

azure-blob-storageazure-storage-explorer
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RinshadR-2100 avatar image
0 Votes"
RinshadR-2100 answered ZacharyLegault-8153 commented

Hi Zachary,


Could you please confirm, as mentioned in the MSFT document https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-s3, did you set the environment variable? This environment variable helps you to authenticate with AWS S3.


Thanks
Rinshad.R

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes, I did set the Env variables. Was getting an entirely different error before realizing I had forgotten that step.

0 Votes 0 ·
SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered SumanthMarigowda-MSFT edited

@ZacharyLegault-8153 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Adding more information to the above response! AzCopy uses an access key, and secret to authenticate with AWS S3. For the destination Blob storage account you can use any of the available authentication options (SAS token, or Azure Active Directory authentication). See here for more information on azcopy aws azure

To copy an S3 bucket to a Blob container, use the following command:

 azcopy cp "https://s3.amazonaws.com/mybucket/" "https://mystorageaccount.blob.core.windows.net/mycontainer<SAS>" --recursive

AzCopy V10 uses the service APIs to read the source data directly into Storage, i.e. the Storage must be able to read the S3 data directly.

Alternatively, if you are looking for a fully managed Platform-as-a-Service (PaaS) option for migrating data from AWS S3 to Azure Storage, consider Azure Data Factory (ADF), which provides more additional benefits: [see here][2]

If the issue still persist, share the commands and screenshot of the error message.

Looking forward for your reply!

Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.