question

AlexBlanes-8504 avatar image
0 Votes"
AlexBlanes-8504 asked SaurabhSharma-msft commented

How to check if any file exists on a folder using Azure File Storage as service link?

Hello,

At the moment I am developing an ETL on azure data factory. The proccess runs well, but in the steps where there are no files to be proccesed. It throws errors. To solve this, I am checkig if there are any files inside the origins folders. In most of the cases, I am able to check if there are any files inside a folder. Because they are inside a blob container of a Storage Account.
166440-imagen.png

But In one case the origin files are on a 'Shared folder' of a blob storage because It is mount as a SFTP. In that case, the unique way I found to acces the files. Where by using a "Azure File Storage" as service link.
166417-imagen.png

With the tests I have made I does not work as the "Azure Data Lake Storage Gen2" where by using '*.csv' on the file path founds any file inside the folder. Instead it returns this error:


Field 'exists' failed with error: 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Azure File operation Failed. Path: originfolder/*.csv. ErrorMessage: Error Message: The specifed resource name contains invalid characters. (ErrorCode: 400, Detail: The specifed resource name contains invalid characters., RequestId: ac72b70d-301a-0001-4757-0d6913000000).,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.StorageException,Message=The specifed resource name contains invalid characters.,Source=Microsoft.Azure.Storage.Common,'.



I hope I made me understand. Thanks in advance

EDIT: This is the method i get the info:
166458-imagen.png
In the next step I just have to access the previous step output.exists

azure-data-factoryazure-files
imagen.png (47.1 KiB)
imagen.png (48.5 KiB)
imagen.png (36.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SaurabhSharma-msft avatar image
1 Vote"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @alexblanes-8504,

Thanks for using Microsoft Q&A !!
Are you talking about sftp feature in Azure Blob Storage or Storage File Shares ?

If you are using a blob container which is hosted as sftp it will appear as a container under Containers like below.
Settings
166991-image.png

Containers section
166981-image.png

So, instead of using a File Storage linked service you can use the a Blob Storage linked service to check for file existence by using a parameterized dataset like below-
167002-image.png
And pass wild card file name from GetMetadata Activity and setting up field list as childitems

167001-image.png

Then you can check the output GetMetadata Activity in an "If Condition" using expression as @not(empty(activity('Get Metadata1').output.childItems)).
167011-image.png
Please let me know if this works for you or in case you have any questions.

Thanks
Saurabh


image.png (73.8 KiB)
image.png (67.3 KiB)
image.png (37.7 KiB)
image.png (36.1 KiB)
image.png (26.9 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @alexblanes-8504,
We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Thanks
Saurabh

1 Vote 1 ·

Hi @SaurabhSharma-msft ,

Sorry for the delay.

Few days ago I understood how it works.

The way I was using in this post was incorrect, because it just checked the existence of the the parent folder.

I ended using the same function as you mentioned in the end of your post.

Thanks

0 Votes 0 ·

Hi @alexblanes-8504,

Great that it works for you. Thanks for the confirmation.


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

Thanks
Saurabh

0 Votes 0 ·