How to skip error and continue unzip process using Azure Data Factory copying activities

Question

I have large zip file that I transferred from SFTP to blob storage. I have created a copy activity to unzip the files from one blob storage (source dataset) to another blob storage (sink dataset). The unzip process has the error : file operation is failed, upload file failed at path:

'***/0153FBA620000B.pdf'.,Source=Microsoft.DataTransfer.Common,''Type=System.ArgumentOutOfRangeException,Message=Number was less than the array's lower bound in the first dimension.

I understood that might because the file it self is compressed using deflate 64 because of its size. Question here:

1. can ADF handle the unzip processing using deflate 64 compression method ?
2. If no, how can I config the ADF copy activity to skip this particular file and continue the unzip process ?

Answer 1

Hi Liu, Faye,

Thanks for reaching out to Microsoft Q&A.

can ADF handle the unzip processing using deflate 64 compression method ?

Yes, you can. Please refer Kranthi Pakkala's answer (dont want to duplicate the answer) on how to perform this step-by-step and configuration setting for both source and sink. Pls check and let me know.

https://stackoverflow.com/questions/57261025/how-to-decompress-a-zip-file-in-azure-data-factory-v2

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 2

Liu, Faye: Yes, you are right. I faced the same issue earlier and was answered as below by the internal team after raising a support ticket with them. I am reproducing my answer here for your reference, hope this will be useful to know what happens internally.

https://learn.microsoft.com/en-us/answers/questions/1265410/what-compression-type-should-we-choose-to-unzip-gz