Azure Data Factory: Cannot save the output of Set Variable into file/Database.

Alireza Tabebordbar 6 Reputation points
2022-04-21T14:21:48.203+00:00

I'm trying to store a list of file names within an Azure Blob container into a SQL db. The pipeline run successfully, but after running the pipeline, it cannot output the values (file names) into the sink database and the table is still empty after the pipeline completed. Followings are the steps I went through to implement the pipeline. I wonder which steps I made mistake.

I have followed the solutions given in the following links:
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy

https://stackoverflow.com/questions/60673220/transfer-the-output-of-set-variable-activity-into-a-json-file-azure-data-fact/61376547?noredirect=1#comment120739615_61376547

Steps: Validating File Exists, Get Files metadata and child items, Iterate the files through a foreach.
1- 195129-1.png

2- Variable defined at the pipeline level to hold the filenames
195119-image.png

3- parameter defined to dynamically specify the dataset directory name
195110-image.png

4- Get Metadata configurations
195199-image.png

5- Foreach settings
@activity('MetaGetFileNames').output.childItems

6 - Foreach Activity overview. A set Variable to set the each filename into the defined variable 'files'. Copy Activity to store the set value into db.
195169-image.png

7- set variable configuration
195205-image.png

8- Copy Activity source configuration. Excel Dataset refers to an empty excel file in azure blob container.
195243-image.png

9- Copy Activity sink configuration
195178-image.png

10-Copy Activity: mapping configuration
195186-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,662 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Alireza Tabebordbar 6 Reputation points
    2022-04-24T02:00:42.517+00:00

    The question is resolved. needs to refer to a file with some dummy values instead of an empty file.
    azure-data-factory-cannot-save-the-output-of-set-variable-into-file-database

    1 person found this answer helpful.