question

AnishaSinghal-2757 avatar image
0 Votes"
AnishaSinghal-2757 asked ChethanReddyR commented

Azure DataFactory: Error while copying data from http source to ADLS Gen2

I am trying to copy the data from an http source to ADLS gen2 and I get below error:

Error: Failure happened on 'Sink' side. ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server:The remote server returned an error: (403) Forbidden.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.,Source=System,'

There is a foreach loop within which the copy from http to ADLS gen2 takes place. This loop works for few records but fails after some time. The configuration settings for the copy data task is as shown below:

90629-image.png

90791-image.png

90774-image.png

90763-image.png


Please let me know if there is a fix for this?


azure-data-factoryazure-data-lake-storage
image.png (38.4 KiB)
image.png (46.2 KiB)
image.png (27.8 KiB)
image.png (40.7 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AnishaSinghal-2757 and welcome to Microsoft Q&A.

To help narrow down the cause, I have a few questions.

  • Is it always the same page that fails?

  • You are copying inside a loop. Does the source change with each iteration, or are you copying the exact same source data each time?

  • Is the loop sequential, or in parallel?

  • How much time passes before it starts to fail?

  • Is there any authentication involved with your http source?

  • The page that fails, can you read it with something other than ADF? (Such as Postman or Curl)


0 Votes 0 ·

@AnishaSinghal-2757 I haven't heard back from you. Are you still facing the issue?

0 Votes 0 ·

@AnishaSinghal-2757 if you found your own solution, please share here with the community.

0 Votes 0 ·

I am facing the same issue while copying data from Gen1 to Gen2 using ADF pipeline. Copy process starts and after copying few files it fails with Forbidden erroron sink side on a file. When u look for the file on Sink (Gen2) the file is copied too.
Not sure what is causing the failure.

Error:

Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Forbidden'. Account: 'xxxx'. FileSystem: 'xx'. Path: 'xxx'. ErrorCode: 'AuthorizationPermissionMismatch'. Message: 'This request is not authorized to perform this operation using this permission.'. RequestId: '69301b22-901f-0009-5125-837848000000'. TimeStamp: 'Tue, 27 Jul 2021 20:26:25 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Operation returned an invalid status code 'Forbidden',Source=,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Forbidden',Source=Microsoft.DataTransfer.ClientLibrary,'

0 Votes 0 ·

Hello @ChethanReddyR and welcome to Microsoft Q&A.

Could you please open a new question thread, and we will help you there.
This thread is old, and your error message seems slightly different.
Thank you.

0 Votes 0 ·

0 Answers