question

PandeyAman-7220 avatar image
0 Votes"
PandeyAman-7220 asked ShaikMaheer-MSFT commented

How to copy HTTP zip file to azure blob

I am running into issue while copying data from http link to azure blob.

HTTP zip file : http://en.www.inegi.org.mx/contenidos/programas/pibo/2013/datosabiertos/eopibt_trimestral_csv.zip

I am able to view the data in the dataset, but getting below mentioned error when running the pipeline.


Operation on target Copy data1 failed: Failure happened on 'Sink' side. ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server:Unable to connect to the remote server,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 200.23.8.41:80,Source=System,'

Please suggest how to resolve this issue

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @PandeyAman-7220 ,

Following up to check is below provided answer helpful? If yes, Please consider clicking on Accept Answer. Accepted answers helps community as well. Thank you.

0 Votes 0 ·

Hi @PandeyAman-7220 ,

Just checking is below provided answer helpful? If yes, Please consider clicking on Accept Answer. Accepted answers helps community as well. Thank you.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @PandeyAman-7220 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

I Copy of Zip file from below link to blob storage using Copy activity. Its working fine. You need to use HTTP Connector source dataset inside copy activity.
HTTP connector is generic to retrieve data from any HTTP endpoint, e.g. to download file.

HTTP zip file : http://en.www.inegi.org.mx/contenidos/programas/pibo/2013/datosabiertos/eopibt_trimestral_csv.zip

Please check below detailed example.

Step1: Copy activity, Source tab HTTP connector type data set. Used "ZipDeflate(.zip)" compression type as our file is .zip type

130743-source.gif

Step2: Copy activity, Sink tab Blob connector type dataset. Used "None" as compression type.

130698-sinkdataset.gif

Step3: Execute pipeline and see results.

130706-execution.gif

Please Note: In above example I unzipped file while loading in to blob storage. If you would like to load .zip file as is in to blob, then use compression type as "none" in both source and sink datasets.

Hope this will help. Please let us know if any further queries.

  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



source.gif (1.3 MiB)
sinkdataset.gif (389.4 KiB)
execution.gif (627.0 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @PandeyAman-7220 ,

Thank you for response. I tried with the same URL what you shared in question. It is working perfectly fine. I feel some settings you might configured incorrectly inside copy activity.

Along with above answer could you please refer below video as well, where similar scenario implemented and follow exactly same steps and see?
https://www.youtube.com/watch?v=EY6Wk5XMvqA

1 Vote 1 ·

I am getting below error now

Error:
Operation on target Copy data2 failed: ErrorCode=UserErrorSourceNotSeekable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Your HttpServer source can't support random read which is requied by current copy activity setting, please create two copy activities to work around it: the first copy activity binary copy your HttpServer source to a staging file store(like Azure Blob, Azure Data Lake, File, etc.), second copy activity copy from the staged file store to your destination with current settings.,Source=Microsoft.DataTransfer.ClientLibrary,'


Zip files conatin two folder with multiple CSV files

0 Votes 0 ·

if I try to copy file as it is to azure blob I am getting the below mentioned error:


Failure happened on 'Sink' side. ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server:Unable to connect to the remote server,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 200.23.8.41:80,Source=System,'

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT PandeyAman-7220 ·

Hi @PandeyAman-7220 ,

This is strange when i tried with exactly same link what you shared its working perfectly fine for me. If you are following all the steps in answer correctly still getting issue then I suggest you to kindly have support ticket for same to deeper analysis.

0 Votes 0 ·