question

Adrian-5124 avatar image
1 Vote"
Adrian-5124 asked ShaikMaheer-MSFT commented

Data Factory Append

Hey!

I am looking to see if there is any way to 'append' data in azure data factory.

127148-image.png



Here is how my data flow looks, each of these datasets have two columns, instead of saving them all into separate files, can I append them together and save into one file?

azure-data-factoryazure-synapse-analyticsazure-databricks
image.png (60.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @Adrian-5124 ,

Thank you for posting query in Microsoft Q&A Platform.

If you are trying to appended columns from different sources then, You can achieve this using Surrogate Key transformation and Join Transformation.

  • Use Surrogate Key transformation on each source data to get unique id columns.

  • After that use Join transformation to join different sources data on the Unique column what we generated above.

    If you are trying to append rows then you can use Union Transformation.

Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Or just add a Union transformation to this data flow above, Union each stream together, and then add your Sink.

0 Votes 0 ·

Thank you @MarkKromer-2402 for inputs on this.

0 Votes 0 ·