question

sivasudarsan-2412 avatar image
0 Votes"
sivasudarsan-2412 asked SaurabhSharma-msft commented

Azure data factory read text file , based on value trigger another pipeline

Adf pipeline activity need to
1) read text file from storage
2) extract value and based on value
3) trigger another pipeline

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

@sivasudarsan-2412 Thanks for using Microsoft Q&A !!

You can use -
1. Lookup activity first to fetch the records and
2. Then use ForEach Activity to traverse through the returned result set
3. Inside For Each Activity you can use the IF Condition activity to execute another pipeline using Execute Pipeline trigger.

Please refer to the below Gif.
80403-adf-readtextfile.gif

Also sharing pipeline code for your reference, if this helps

 {
     "name": "pl_read_text_trigger_childPipeline",
     "properties": {
         "activities": [
             {
                 "name": "Lookup1",
                 "type": "Lookup",
                 "dependsOn": [],
                 "policy": {
                     "timeout": "7.00:00:00",
                     "retry": 0,
                     "retryIntervalInSeconds": 30,
                     "secureOutput": false,
                     "secureInput": false
                 },
                 "userProperties": [],
                 "typeProperties": {
                     "source": {
                         "type": "DelimitedTextSource",
                         "storeSettings": {
                             "type": "AzureBlobStorageReadSettings",
                             "recursive": true,
                             "enablePartitionDiscovery": false
                         },
                         "formatSettings": {
                             "type": "DelimitedTextReadSettings"
                         }
                     },
                     "dataset": {
                         "referenceName": "txtDestination",
                         "type": "DatasetReference"
                     },
                     "firstRowOnly": false
                 }
             },
             {
                 "name": "ForEach1",
                 "type": "ForEach",
                 "dependsOn": [
                     {
                         "activity": "Lookup1",
                         "dependencyConditions": [
                             "Succeeded"
                         ]
                     }
                 ],
                 "userProperties": [],
                 "typeProperties": {
                     "items": {
                         "value": "@activity('Lookup1').output.value",
                         "type": "Expression"
                     },
                     "activities": [
                         {
                             "name": "If Condition1",
                             "type": "IfCondition",
                             "dependsOn": [],
                             "userProperties": [],
                             "typeProperties": {
                                 "expression": {
                                     "value": "@equals(item().ID,'3')",
                                     "type": "Expression"
                                 },
                                 "ifTrueActivities": [
                                     {
                                         "name": "Execute Pipeline1",
                                         "type": "ExecutePipeline",
                                         "dependsOn": [],
                                         "userProperties": [],
                                         "typeProperties": {
                                             "pipeline": {
                                                 "referenceName": "pl_childPipeline",
                                                 "type": "PipelineReference"
                                             },
                                             "waitOnCompletion": true,
                                             "parameters": {
                                                 "waitTime": {
                                                     "value": "3",
                                                     "type": "Expression"
                                                 }
                                             }
                                         }
                                     }
                                 ]
                             }
                         }
                     ]
                 }
             }
         ],
         "annotations": []
     }
 }


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@sivasudarsan-2412 Please let me know if you find above reply useful. If yes, do click on 'Mark as answer' link in above reply. This will help other community members facing similar query to refer to this solution. Thanks.

0 Votes 0 ·