question

WilliamsDamianBarrachd-7717 avatar image
0 Votes"
WilliamsDamianBarrachd-7717 asked WilliamsDamianBarrachd-7717 answered

Passing an object as a parameter to a Data Factory pipeline

Hi Team,

I'm trying to use dynamic mapping in a pipeline Copy activity. I want to read the mapping from a SQL database using a Lookup, then send it to the pipeline.

I've created the pipeline and a parameter of type Object, and it works when I manually paste in the string below (Working String) as a value for the for the parameter.

If I then create another pipeline with an activity to call this pipeline and paste in the same string for the parameter value, it doesn't work. When I examine the sub-pipeline's input, I can see that all the double-quotes have been excaped with a backslash, which then doesn't seem to parse correctly. I get the same results if I try and generate my string programatically via a Lookup, either as an acitivty in the same pipeline, or from a calling pipeline.

I've also tried generating the string with | instead of ", then in the mapping section of my Copy activity, using a replace to turn them all back to ", but this doesn't work either.

Is there a workaround?

Thanks

Damian.

Working String
{ "type": "TabularTranslator", "mappings": [{ "source": { "path": "['ExternalID']" }, "sink": { "name": "ExternalID", "type": "String" } }, { "source": { "path": "['Name']" }, "sink": { "name": "Name", "type": "String" } }, { "source": { "path": "['Description']" }, "sink": { "name": "Description", "type": "String" } }, { "source": { "path": "['SessionType']" }, "sink": { "name": "SessionType", "type": "String" } }, { "source": { "path": "['StartDateTime']" }, "sink": { "name": "StartDateTime", "type": "DateTime" } }, { "source": { "path": "['EndDateTime']" }, "sink": { "name": "EndDateTime", "type": "DateTime" } }, { "source": { "path": "['ExpectedMark']" }, "sink": { "name": "ExpectedMark", "type": "String" } }, { "source": { "path": "['Organiser']" }, "sink": { "name": "Organiser", "type": "String" } }, { "source": { "path": "$['OrganisationID']" }, "sink": { "name": "OrganisationID", "type": "String" } }, { "source": { "path": "['Tags']" }, "sink": { "name": "Tags", "type": "String" } }], "collectionReference": "$['value']", "mapComplexValuesToString": true }


azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @WilliamsDamianBarrachd-7717,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Yes, I am able to reproduce the issue. I will reach out to Product team for more details and get back to you with updates.

Thanks for your patience!

1 Vote 1 ·
HarithaMaddi-MSFT avatar image
1 Vote"
HarithaMaddi-MSFT answered HarithaMaddi-MSFT commented

Hi @WilliamsDamianBarrachd-7717,

I looked at it closely and observed that the when output from lookup is passed it is not able to treat JSON correctly. I have used below casting to the parameter in mapping section that enabled copy activity to read the value correctly as shown below.

 @json(string(pipeline().parameters.mapping))

Below is GIF showing the details

37531-dynamicmappingparentpipelineadf.gif

Please suggest if this addresses the requirement, let us know for further queries and we will be glad to assist.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HarithaMaddi-MSFT avatar image HarithaMaddi-MSFT WilliamsDamianBarrachd-7717 ·

@WilliamsDamianBarrachd-7717 - Glad to know it worked.

0 Votes 0 ·
BuddhabhattiAlpa-8144 avatar image
0 Votes"
BuddhabhattiAlpa-8144 answered BuddhabhattiAlpa-8144 edited

Hi Can you please let me know your source and sink setting of copy activity here.
Also can I pass append variable as copy activity source.?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

WilliamsDamianBarrachd-7717 avatar image
0 Votes"
WilliamsDamianBarrachd-7717 answered

Source and Sink were both SQL Server databases, with parameterised table names. I haven't tried using a variable as the source, but it should work if you put parameters on the Linked Service / Dataset and populate them using the variable.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.