question

AlirezaTabebordbar-0580 avatar image
0 Votes"
AlirezaTabebordbar-0580 asked PRADEEPCHEEKATLA-MSFT commented

Azure Data Factory: Cannot save the output of Set Variable into file/Database.

I'm trying to store a list of file names within an Azure Blob container into a SQL db. The pipeline run successfully, but after running the pipeline, it cannot output the values (file names) into the sink database and the table is still empty after the pipeline completed. Followings are the steps I went through to implement the pipeline. I wonder which steps I made mistake.

I have followed the solutions given in the following links:
https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy

https://stackoverflow.com/questions/60673220/transfer-the-output-of-set-variable-activity-into-a-json-file-azure-data-fact/61376547?noredirect=1#comment120739615_61376547

Steps: Validating File Exists, Get Files metadata and child items, Iterate the files through a foreach.
1- 195129-1.png


2- Variable defined at the pipeline level to hold the filenames
195119-image.png

3- parameter defined to dynamically specify the dataset directory name
195110-image.png

4- Get Metadata configurations
195199-image.png

5- Foreach settings
@activity('MetaGetFileNames').output.childItems

6 - Foreach Activity overview. A set Variable to set the each filename into the defined variable 'files'. Copy Activity to store the set value into db.
195169-image.png

7- set variable configuration
195205-image.png

8- Copy Activity source configuration. Excel Dataset refers to an empty excel file in azure blob container.
195243-image.png

9- Copy Activity sink configuration
195178-image.png

10-Copy Activity: mapping configuration
195186-image.png


azure-data-factory
1.png (19.7 KiB)
image.png (11.5 KiB)
image.png (11.6 KiB)
image.png (24.9 KiB)
image.png (22.8 KiB)
image.png (7.7 KiB)
image.png (50.9 KiB)
image.png (46.7 KiB)
image.png (41.3 KiB)
image.png (46.1 KiB)
image.png (37.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

AlirezaTabebordbar-0580 avatar image
1 Vote"
AlirezaTabebordbar-0580 answered PRADEEPCHEEKATLA-MSFT commented

The question is resolved. needs to refer to a file with some dummy values instead of an empty file.
azure-data-factory-cannot-save-the-output-of-set-variable-into-file-database



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @AlirezaTabebordbar-0580,

Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

0 Votes 0 ·