question

MohsenAkhavan avatar image
0 Votes"
MohsenAkhavan asked ShaikMaheer-MSFT commented

I need help about design data pipline or data flow in Azure Data Factory

I'm new to Azure Data Factory and I want to create an Azure Data Factory for the below scenario.
I have a Postgres database and add data every minute.
1. I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).
2. Then, run some manipulation (function) on the output of step1.
3. The output of step 2 should be sent to a database or a topic of service bus.

I need some help with which activity and solution I should use.



azure-data-factory
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MohsenAkhavan ,

Just checking in to see if the below answer provided helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @MohsenAkhavan ,

Following up to check is provided answer works for you. If yes, Please accept answer. Feel free to post if any further queries. Thank you.

0 Votes 0 ·

Hi @MohsenAkhavan ,

Following up to check is provided answer helped you. Please Accept Answer and Up-Vote. If any further queries do let us know

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
1 Vote"
ShaikMaheer-MSFT answered

Hi @MohsenAkhavan ,

Thank you for posting query in Microsoft Q&A Platform. Below is the details.

I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).

Copy Activity helps to perform data movement from source to sink without any transformation or manipulation. You have to use Query option so that you can take out only selected rows. But you mentioned selected rows of your source data should first undergo transformations. Hence You should go with Data Flows to implement scenario.
Copy Activity - https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview

Then, run some manipulation (function) on the output of step1.

Data flow source transformation helps to run query on your source to get selected data

107989-surcetrans.png

Source Transformation - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-source

The output of step 2 should be sent to a database or a topic of service bus.

You can use Sink transformation in dataflow to load transformed data in to different storages. Please check below link for supported storages.
Sink Transformation - https://docs.microsoft.com/en-us/azure/data-factory/data-flow-sink

Service bus cannot be used as Sink in Dataflows, Hence workaround would first load data in supported storages using dataflows and from there load it to Service bus if require.

Hope this information is helpful. Thank you.




surcetrans.png (54.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.